﻿Science-Based Medicine Reviews: Critical Thinking 

A Critical Guide for the Health Practitioner and Consumer 
2025 Edition

Executive Editor Steven Novella MD
Managing Editor David Gorski MD PhD
Series Editor Mark Crislip MD

   Introduction 
Steven Novella
The world of health care is becoming an increasingly complex and challenging place. The internet is changing the way we generate and access information, and health information is one of the most popular uses of the internet. It represents an unprecedented democratization of information and opinion. 
While there are many advantages and benefits to such information access, there are also many challenges to health professionals, regulators, academics, and consumers. There is a proliferation of health related products and services, with many provocative health claims. Regulations are generally not keeping up, and often have been watered down, in the face of this onslaught of claims and products. 
The consumer is left with an overwhelming array of health claims, many of which challenge and even directly oppose the traditional experts and institutions. 
The mission of Science-Based Medicine (SBM) is to be a source of reliable information and analysis to help navigate the wild west of health information on the internet and elsewhere. The authors who contribute to this series of e-books are all physicians and other health care professionals who have a great deal of experience educating the public about complex medical issues. We especially take on controversial issues and try to expose pseudoscience and fraud wherever possible.
The Philosophy of Science-Based Medicine
The philosophy of SBM, at its core, is simple: Safe and effective health care is critical to everyone’s quality of life; so much so that it is generally considered a basic human right. Therefore, health care products and services that work and are safe are better than those that do not work or are unsafe. 
The best method for determining which interventions and health products are safe and effective is, without question, good science. Therefore it is in everyone’s best interest for health care to be systematically evaluated by the best science available.
Too often the nature of science itself is misunderstood or misrepresented to the public. Science is not an arcane and privileged discipline. By its very nature it is meant to be transparent and public. Science is nothing more than a systematic and careful use of evidence and logic to evaluate factual claims. And good science possesses certain virtues that are not unique to science but generic to all intellectual endeavors: fairly accounting for all available evidence, using valid and internally consistent logic, using unambiguous concepts and language, proper use of statistics, being quantitatively precise and accurate, and above all being honest.
And yet there are numerous and powerful influences in society that strongly appose the scientific basis of medicine. Driven by some combination of ideology or the desire for profit they wish to eliminate standards of science in health care, or (often under the guise of “health care freedom”) create a double standard in which unscientific methods and products can thrive unchecked. Others simply lack the training or knowledge to achieve minimal standards of quality for scientific medicine. And even the best traditions of scientific medicine can benefit from more critical analysis.
Recognition of the need for careful scientific analysis in medicine partly stems from a recognition that it is very difficult (almost impossible) to reliably assess treatments anecdotally. Human perception, memory, and cognition are prone to a host of flaws, weaknesses, and biases that will tend to mislead us, especially towards conclusions that we wish to be true. Placebo effects, confirmation bias, and simple statistics can make any intervention seem effective, even when they are worthless or harmful. 
Science-Based Medicine vs Evidence-Based Medicine
Within the practice of medicine there is already a recognition of the need to raise the standards of evidence and the availability of the best evidence to the practitioner and the consumer – formalized in the movement known as evidence-based medicine (EBM). 
EBM is a vital and positive influence on the practice of medicine, but it has its limitations. EBM tends to focus on clinical trial results to the exclusion of scientific plausibility. The focus on trial results (which, in the EBM lexicon, is what is meant by “evidence”) has its utility, but fails to properly deal with medical modalities that are highly implausible, or seem to defy well-established findings in science.
All of science describes the same reality, and therefore it must (if it is functioning properly) all be mutually compatible. Collectively, science builds one cumulative model of the natural world. This means we can make rational judgments about what is likely to be true based upon what is already well established. This does not necessarily equate to rejecting new ideas out-of-hand, but rather to adjusting the threshold of evidence required to establish a new claim based upon the prior scientific plausibility of the new claim. Failure to do so leads to conclusions and recommendations that are not reliable, and therefore medical practices that are probably not safe and effective.
The SBM Series
This is why the authors of this series strongly advocate for science based medicine – the use of the best scientific evidence available, in the light of our cumulative scientific knowledge from all relevant disciplines, in evaluating health claims, practices, and products. 
Some of the articles here will analyze individual studies, and explore the strengths and weaknesses of their methods, separating what the study really tells us from how it may be represented in the media. Other articles will summarize what many studies tell us about a specific medical question. 
We will also explore health information on the internet, separating out which information is reliable from that which is misleading, erroneous, or possibly even fraudulent. Likewise we will critique mainstream media coverage of health topics, giving the reader specific tools for to critically analyze health information for themselves.
Other articles will explore topics related to health regulation and academia. Regulations have struggled to keep up with the rapidly changing health care market place, with critical gaps that leave the health care consumer responsible for their own protection. Academic institutions themselves, long the defenders of high standards of scholarship in all fields, including medicine, have been infiltrated by promoters of pseudoscientific belief systems. The result is that there is no longer any single reliable guarantee of scientific integrity. 
The SBM series of e-books is organized by topic, such as acupuncture, chiropractic, homeopathy, and many others. The books are collections of essays, not intended to be a systematic reference, but rather to explore the various scientific issues related to each topic. The goal is to give the reader critical thinking skills and a deeper understanding of, not just the current findings of science, but the process of science, specifically how it relates to health care and the topic of each book. 
Science-based medicine can serve as your pocket guide through the wilderness of health information.

As of 2025, there are over 5000 essays on the Science Based Medicine website. I do no know about you, but I find it difficult to find any post I wrote, much less one written by another contributor. 

So I have updated the books and include almost all the articles through 2025. The volumes are

CAM Concepts
Chiropractic
COVID 
Critical Thinking
Diet and Nutrition
Ethics and CAM
Herbs, Supplements, and Diet
Homeopathy
Infections and Immunity
Law and CAM
Media and Internet
Medicine, Public Health, Dentistry and Pediatrics
Miscellaneous CAM
Naturopathy
Neurology and Psychiatry
Placebo
Quackademic Medicine
Regulation. FDA and FTC
SBM or EBM
Stem Cells, Genetics, AI and Gender
Supplements, Vitamins and Herbs
Vaccine & Anivax
Vaccines and Autism
Vaccines and Preventable Infections

The articles are roughly organized by publication date and topic. Many articles will be in multiple volumes as an essay of, for example, chiropractic and law will be found in both the chiropractic volume and the law volume.

You may find the organization a bit odd. I do. But 5000 plus articles are difficult to organize and I have more to do in my retirement than work of these volumes.

Still, if you find I have missed a nentry or some glaring error, drop me a note and I will fix it for the next edition.

About the authors
Steven P. Novella, MD – Founder and Executive Editor Science Based Medicine.
Founder and currently Executive Editor of Science-Based Medicine Steven Novella, MD.
David H. Gorski, MD, PhD – Managing Editor. Science Based Medicine.
Mark A. Crislip, MD. Editor, Science Based Medicine Guides. 
He did he editing of all the volumes so is to blame for any mistakes.
As to all the other contributors, if you are interested click on the ‘Link’ at top of each essay to go the SBM site, where further information about each author can be found.
A Note On References

The chapters in this book were originally blog entries at ScienceBasedMedicine.org.
Blog and web pages lend themselves to hyperlinks to sources and references that are tedious to translate into footnotes for this ebook.
Under the title of each essay is a hyperlink, cleverly titled ‘Link’, to the original blog entry with all the hyperlinks and references, as well as the comments to the blogs, which often have interesting and thoughtful insights by our readers, as well as some curious comments by the not so thoughtful.
Graphs, tables and links to videos are not in included these essays due to issues of copyright and permission. However, thanks to the ‘Link’, the graphics are a click away.
Thanks for reading the site.
Mark Crislip 






   Overview
There is what to think and there is how to think.
Science and medicine are constantly changing, being refined and evolving as knowedge advances. 
The essays in this collection concern how to think about issues in science based medicine, from the reality behind the placebo effect to how to read the medical literature. It is a primer on how to think and how to avoid thinking badly.

In Memoriam – Dr. Harriet Hall
Remembering our colleague and friend.
Steven Novella on January 15, 2023
Link
As we previously announced, co-founder, editor, and contributing author of SBM, Dr. Harriet Hall, unexpectedly passed away on the evening of January 11. This is a tremendous loss for SBM, for the skeptical and science-based medicine community, and for us personally. Great appreciation to her husband, Kirk, who thoughtfully provided a short bio for Harriet:
Harriet Anne Hall was born to Albert L Hoag and Edna (Barnes) Hoag in St. Louis, Missouri on 2 July 1945. She was the eldest of four children (the twins Steven & Stephanie and the youngest Andrea). Harriet attended the University of Washington as an undergraduate, where her father taught and graduated with her medical degree from there in 1970. She was commissioned First Lieutenant in the US Air Force on 13 June 1970 and completed her Rotating Internship at the David Grant USAF Medical Center in 1971. In her early AF career, she was a General Medical Officer and was stationed in Spain for seven years. After her return to the US Harriet attended the Aerospace Medicine Primary Course to become a Flight Surgeon, graduating in 1979. In the same year Harriet became Certified by the American Board of Family Practice. She was assigned to Francis E. Warren AFB where she met and married Kirk Albert Hall, Jr. Harriet retired as a Full Colonel at McChord AFB, WA. She lived in Puyallup, Washington until her death on 11 Jan 2023.
But perhaps the greatest impact Harriet had was after she retired from her first career and essentially took up a second career as a science communicator. I first met Harriet about 20 years ago, at a science and medicine conference hosted by CSI. We immediately connected, as two physicians highly skeptical of the claims and practices loosely collected under the umbrella of “alternative medicine”. We worked together, mostly through Wallace Sampson (another founding contributor to SBM whom we lost a few years ago) and his work with CSI, including the journal, The Scientific Review of Alternative Medicine. When I decided to begin my own project here at SBM, Harriet was an obvious choice to join the effort.
Since then Harriet has been tireless in her efforts to help educate the public about how science should inform the practice of medicine, and how medicine, and even common sense, can go horribly wrong when we abandon good science as our guide. Among her many contributions, Harriet is remembered for coining the phrase, Tooth Fairy Science, which nicely crystalized and communicated one of the many core problems with alternative science. To paraphrase, you can study in detail all of the aspects of the Tooth Fairy phenomenon without ever getting to the core question – the only question that really matters – does the Tooth Fairy actually exist?
In addition to being an editor and regular contributor to SBM, Harriet also ran her own newsletter, SkepDoc, and contributed to Skeptic magazine, Skeptical Inquirer magazine, and O, The Oprah Magazine. She would lend her time and efforts to many projects promoting science and skepticism, including Quackwatch, CSI, The Skeptic’s Toolbox, and the Institute for Science in Medicine. She worked through her retirement, even when her health began to fail her, right up until the end.
Harriet also published two books. The first, Women Aren’t Supposed to Fly: The Memoirs of a Female Flight Surgeon, chronicles the challenges she faced in a male-dominated career. And just this year she published There’s No Such Thing as the Tooth Fairy! This is an illustrated children’s book that will hopefully introduce a generation of children to skeptical thinking.
Unfortunately, during the last few years Harriet suffered from declining health, including heart failure and arrhythmias. These health issues limited her ability to travel, but also coincided with the pandemic, which greatly limited live events for over two years. This means I had not seen Harriet in person for several years, which I regret. But through it all she never missed a deadline.
We will certainly miss Harriet’s contributions here. She was always fearless and tireless in promoting and defending science, reason, and above all what she considered the best interest of her patients. In her second career she made the entire world her patient. She was driven by the desire to protect the public from misinformation, exploitation, and the bad science and thinking that lead to bad medicine. That is a life worth celebrating.

   Resistance is futile? Hell, no! (A call to arms)
David Gorski
Link
Well, I won’t back down 
No, I won’t back down 
You can stand me up at the gates of hell 
But I won’t back down 
Gonna stand my ground 
Won’t be turned around 
And I’ll keep this world from draggin’ me down 
Gonna stand my ground 
And I won’t back down 
From “I Won’t Back Down” by Tom Petty, 1989
This week, in a little bit of a departure, I have a minor bone to pick with our fearless leader and his podcast partner in crime Rebecca Watson (a.k.a. the Skepchick), who both managed to annoy me a bit the other day. (Don’t worry, Steve and Rebecca, I still love you guys…)
I’ll explain. You see, I had originally had a much different topic in mind for this week. Indeed, I even had my post mostly written by Saturday morning, when I had to take care of some mundane personal business, namely getting an oil change and some minor work done on my car. Since I need my car to commute to work and the maintenance needed was relatively minor, I decided to wait for the work to be done. As is my wont when sitting in waiting rooms with nothing much else to do, I decided to plug my earphones into my iPhone and catch up on some podcasts. Since the dealer also had free wifi, I brought my laptop along as well, the better to finish up my originally intended post.
The first thing I realized as I perused the list of unlistened-to podcasts was that I had fallen far behind in listening to one of my favorite podcasts, The Skeptics’ Guide to the Universe. To begin catching up, I decided to start with what was at the time the most recently available episode, specifically the July 9 podcast, figuring I could work my way back to through the earlier ones and thereby catch up with at least two episodes before my car was ready. In the second segment (beginning around 14:31 minutes into the podcast), Steve Novella and crew discussed a bit the recent news that the National Institute of Mental Health was trying to resurrect a dubious and highly unethical clinical trial proposed to test chelation therapy as a treatment for autism, referencing his excellent post on this very blog about why the trial is scientifically dubious (at best) and totally unethical. So far, so good.
Then the conversation veered into another area that I agree with, namely the utter uselessness of National Center for Complementary and Alternative Medicine (NCCAM) and how its main purpose is more proselytization for “alternative” and “complementary” medicine than actual rigorous scientific research, as I pointed out before in one of my earliest posts for SBM. As Steve pointed out that, for all the hundreds of millions of dollars spent by NCCAM, not a single new medicine or treatment has been added to the armamentarium of modern medicine, nor, even more importantly, have CAM practitioners abandoned a single bit of unscientific medicine due to any of the negative studies. Indeed, their enthusiasm hasn’t been dampened in the least. This line of discussion led to the question of whether we, as skeptics and advocates of science- and evidence-based medicine need to rethink and refocus our efforts.
It was at this point that Steve pointed out that there are quite a few of us dedicated to science-based medicine who are a bit pessimistic, using my post, which I dubbed, tongue-in-cheek of course, Resistance Is Futile. Steve and Rebecca’s reaction to my post made me think that perhaps I should be rethinking more than just how I’m going to contribute to promoting science-based medicine and opposing the infiltration of pseudoscience into academia, because apparently Steve, at least, interpreted my post as though I were saying that the whole fight is pointless and that we might as well give up. (Rebecca, I think, didn’t actually read the article, but she should.) In any case, Steve seemed to think my article was a perfect representation of this pessimistic attitude, which Rebecca characterized as as a “useless attitude to have,” a sentiment to which Steve agreed.
I agree that’s a useless attitude to have. Of course, if that were truly my attitude, I would have politely informed Steve long ago that I no longer feel it’s worth my while to contribute to this blog. Yet here I remain, as cantankerous and histrionic as ever, and my other online activities against quackery and antiscience continue apace. If that were truly my attitude, I would no longer bother to write letters, open or otherwise, to station managers about “investigative reporting” that is no more than parroting of anti-vaccine propaganda. I would not take the chance that such letter-writing would come to the attention of the public relations director at my institution. If such were truly my attitude, I would not spend so much time reading and learning about CAM in order to understand it well enough to understand what is useless (the vast majority of it) and what little gems might be found in it.
After all, if it is true that “resistance is futile” and we have already lost, why bother? Better just to hunker down, keep my head down, save what can be saved, and avoid making too many waves.
In retrospect, I suspect that perhaps I didn’t make the tongue-in-cheek aspect of my post clear enough. As regular readers of SBM know, I’m usually not particularly subtle, and I didn’t think I was subtle with my post. However, I did give my geek side free rein, complete with allusions to Star Trek: The Next Generation, the Borg, and Locutus of Borg. It makes me wonder if Steve failed to recognize my loving appropriation of other direct quotes other than the famous “resistance is futile” from Star Trek: The Next Generation‘s best villains of all time, the Borg, slightly altered in order to make my point. When I said, “The Brave(well) New World is upon us. Resistance is futile. Your life as it has been is over. From this time forward, medical culture will adapt to service CAM. Science is irrelevant,” I was actually having a bit of fun making my point, as Star Trek fans probably recognized. Not enough, apparently. Perhaps next time I’ll put a smiley emoticon after it.
Sadly, of course, there are many reasons to be pessimistic. There’s no way of getting around it. By any measure you care to name, scientific medicine is presently losing the war against quackery at every level. Advocates of scientific medicine are outnumbered, outfunded, and outmaneuvered, while the vast majority of medicine and–this is what really galls me–even academic medical faculty remains apathetic. Consider:
1) As both Wally Sampson and Kimball Atwood have pointed out, advocates of unscientific medicine have cleverly coopted the very language of science and medicine, turning it against those who would resist the infiltration of pseudoscience, leading to Wally’s lament over how accepting we as the medical profession have become towards pseudoscience. What once was “quackery” or “pseudoscience” is now merely “unconventional” or “complementary,” and extreme lack of scientific plausibility for a treatment modality seems no longer to be a barrier to clinical trials.
2) Through the passage of the DSHEA of 1994, the federal government has hobbled the FDA and in essence given free rein to supplement manufacturers to make almost any dubious health claim they wish, as long as the claim is sufficiently vague. Meanwhile legislators such as Dan Burton and Ron Paul lobby aggressively for “health freedom” laws that would in essence inoculate quackery from prosecution and allow quacks to operate without any pesky interference from the FDA or FTC.
3) Due to high profile cases where state child welfare agencies have attempted to strip parents of custody of children with cancer to prevent them from choosing quackery over scientific medicine for their child, ill-advised laws, such as “Abraham’s Law” in Virginia, have been passed that in effect strip protections against quackery from teens and older children unfortunate enough to have parents who believe in unscientific medicine.
4) As I and others have pointed out, quackery has infiltrated academic medicine to such an extent that not only are many of our major academic medical centers dedicating divisions, centers, or departments to what Dr. R. W. Donnell has termed “quackademic medicine,” but at least one has required it as part of the mandatory medical school curriculum from day one, bringing a whole new meaning to the term “integrative” medicine, as it integrates scientific medicine with pseudoscientific quackery.
5) Meanwhile, Dr. Andrew Weil has succeeded in “integrating” pseudoscience into the eight family practice residencies, with more virtually guaranteed to follow, and Dr. David Katz at Yale has constructed the foundation for an academic empire of woo based on a more “fluid” concept of evidence.
6) The Bravewell Collaborative has many millions of dollars to spend promoting unscientific medicine to any medical school or academic medical center willing to receive its largesse.
7) NCCAM has a budget of over $120 million a year and has spent over $1 billion so far researching, but also promoting CAM. Worse, it is funding trials that are not only scientifically dubious but arguably downright unethical; for example, the TACT trial of chelation therapy for atherosclerotic coronary artery disease, the trial of the “Gonzalez regimen” for pancreatic cancer, and the proposed trial of chelation therapy for autism. Adding an approximately equal money to the pot of CAM funding is the National Cancer Institute, which funds over $120 million a year worth of CAM research related to cancer.
8) Legislatures are passing laws that not only recognize unscientific “healing” traditions as legitimate professions but some states have even mandated that health insurance plans pay for such “treatments.”
9) Anti-vaccine activists are sowing fear and distrust over vaccines, even going so far as to march on Washington. As a result, in the U.K. measles, once thought conquered, has become endemic again, and in the U.S. measles outbreaks are increasingly common.
10) Our government is making deals with China to study “traditional Chinese medicine,” possibly as a quid pro quo for the Chinese granting the FDA inspection stations in Chinese cities, possibly as an attempt to control adulteration of Chinese herbal medicines.
Yes, folks, it’s bad out there in medicine right now. Real bad. Horribly, bad. So bad that even our ever optimistic leader Steve recently agreed when Quackwatch webmaster Stephen Barrett characterized the era we live in as the “golden age of quackery and antiscience.” There’s nothing to be gained for us by pretending otherwise other than perhaps the happy self-delusion of ignorance, and we have to face facts: The situation isn’t going to get better any time soon, no matter what we do. In fact, it very well may continue to get worse for the foreseeable future. That’s not being “pessimistic.” It’s being realistic. The forces arrayed against scientific medicine are strong, well-funded, and utterly ruthless about forwarding their agenda, the “integration” of antiscience with science. Maybe resistance is futile. Maybe science is irrelevant. Maybe scientific medicine will be forced to adapt to serve CAM. Maybe Rebecca and Steve characterized my attitude correctly. Maybe we advocates of science-based medicine have already lost the struggle.
To which I reply: Hell, no! Or, to echo Rebecca Watson’s words right back at her, “Screw that!”
I can speak for no one but myself, but, as Steve said in his podcast, while breath remains in me and I can still do something, I will not give up, even though I do not anticipate anything resembling victory in the few decades I’m likely to have left on this earth. The stakes are too high, and future generations will not judge us kindly if we surrender to antiscience with a whimper. In this age of seeming science, where the human genome has been sequenced, where we understand more than at any time in history about how the body works and where things break down to result in disease, it is a bitter irony that it is at this very same time that academic medicine is retreating to therapies based on pre-scientific concepts of how nature and the human body work. Thanks to unprecedented scientific advances in our understanding of disease, we stand at the threshold of what could be a golden age of medicine and surgery. It is profoundly disturbing to me that it could be reasonably characterized by anybody as a “golden age of quackery.”
I happen to be a World War II buff and have been ever since I was around 12 or so. As a teenager, I used to devour books on World War II history. When I was a teenager I had a simplistic understanding of the war, but as I got older I realized that the situation was always more complex than the simple narrative that we are taught in school. I also now realize that using World War II analogies is fraught with danger in that it’s hard not to fall prey to Godwin’s Law. However, I point out that World War II remains a good analogy for not giving up even when things seem utterly beyond redemption.
Think of the world in May 1942, six months after Pearl Harbor. Put yourself in the Allies’ place. Europe was in the hands of the enemy, who was rapidly advancing east into Russia, having only been stopped within sight of the minarets of the Kremlin, only to advance on new fronts to the south in the spring. In the Pacific, Japanese forces had driven U.S forces out of Guam, the Philippines, Wake Island, and virtually the entire Pacific west of Hawaii other than Midway Island and Australia. By any measure, the Allies were in deep, deep trouble. I imagine myself alive at that time, beset by a constant barrage of bad news, of loss after loss and retreat after retreat. How could anyone alive at that time believe that the Allies would eventually not just prevail but utterly destroy their enemies less than four years hence? Or picture yourself in London in the fall of 1940, with wave after wave of bombers hitting your fair city and other cities in Britain, pushing the RAF to the brink of destruction fending them off. How could any Londoner alive in late 1940 ever envision total victory less than five years hence?
Of course, the battle we are in is not World War II and CAM advocates are certainly not Nazis (with the possible exception of Ernst Zündel and Dr. Lorraine Day; every movement has its bad apples, I guess), but in terms of whether medicine continues to strengthen its fragile commitment to a scientific basis for its treatments or retreats to the age of Samuel Hahnemann it is every bit as apocalyptic. In terms of the strategic situation, we advocates of science-based medicine are the Allies in 1942, except that we cannot count on a brief battle during four days in June near Midway Island to turn the tide of the war. There will be no sudden reversals. Our task is more akin to the “island hopping” that the U.S. Marines undertook in the Pacific, in which Marines starting with Guadalcanal had to take one little island after another away from the Japanese, than it is to D-Day.
And, yet, despite how badly we are outgunned, outnumbered, and out-spent, there is reason for hope. It is at the moment but a glimmer, but if we can convince enough physicians that there is a serious problem we may have a chance. Particularly heartening to me is that in the last year we have seen something I cannot recall ever seeing before: three major books on alternative medicine that do not represent feel-good, credulous acceptance of the tenets of unscientific medicine. They are not books whose authors you are likely ever to see on The Oprah Winfrey Show any time soon. No, rather they are hard-nosed, skeptical, and they take no prisoners in describing why the vast majority of so-called CAM is nothing more than a demonstration of the power of the placebo effect and all-too-human cognitive shortcomings that all humans, doctors and patients alike, have. I’m referring, of course, to R. Barker Bausell’s Snake Oil Science: The Truth about Complementary and Alternative Medicine; Simon Singh’s and Edzard Ernst’s Trick or Treatment: Alternative Medicine on Trial; and Rose Shapiro’s Suckers: How Alternative Medicine Makes Fools of Us All. Moreover, as the popularity of CAM grows and as it infiltrates “conventional” medicine more and more, inevitably its profile will become high enough that even the apathetic physicians may become alarmed as stories of patients who come to harm through quackery come to light. There will be a backlash; indeed, the backlash may have already started. For example, just last month, the FDA, roused out of its torpor for reasons known only to its leaders but nonetheless fortuitously, started an aggressive campaign to stop the selling of fraudulent cancer “cures.” Meanwhile, there is a growing and vibrant movement in the medical and scientific blogosphere to counter the pseudoscience that makes up so much of “alternative medicine.” Indeed, that may be our last, best hope for ultimately prevailing. All is not lost, but it could be if we allow ourselves to fall into complacency again.
Of course, what the Allies had in World War II were leaders who would not give up, who would not surrender, no matter how hopeless the situation seemed. There was Winston Churchill saying in the darkest depths of the war after the fall of France and the retreat at Dunkirk:
We shall go on to the end, we shall fight in France, we shall fight on the seas and oceans, we shall fight with growing confidence and growing strength in the air, we shall defend our Island, whatever the cost may be, we shall fight on the beaches, we shall fight on the landing grounds, we shall fight in the fields and in the streets, we shall fight in the hills; we shall never surrender, and even if, which I do not for a moment believe, this Island or a large part of it were subjugated and starving, then our Empire beyond the seas, armed and guarded by the British Fleet, would carry on the struggle, until, in God’s good time, the New World, with all its power and might, steps forth to the rescue and the liberation of the Old.
Where is science-based medicine’s Winston Churchill?
Speaking of World War II analogies, I’d even settle for science-based medicine’s Bluto Blutarski (from one of the greatest comedies of all time, Animal House) asking, “Was it over when the Germans bombed Pearl Harbor?” and confidently rallying the troops by bellowing, “Nothing is over until we decide it is”:
Was it over when the Germans bombed Pearl Harbor
How’s that for a more accessible pop culture reference than the Borg? That scene from Animal House always makes me laugh. In this context, I can’t help but think that an SBM Deathmobile would be an über-cool thing to have, perhaps with the head from a statue of Samuel Hahnemann bolted to its hood instead than the head of Faber, as was used in the movie. The humor aside, the scene above also has a very cogent message for us: No matter how bad things are, never give up. There is always something that can be done, although in our current predicament as advocates of scientific medicine what should be done probably doesn’t involve armor-plating a car, painting it black with skulls on it, and crashing it into the grandstand at a parade. Bummer.
I realize that I’ve not only painted a very dire picture but probably also gone completely off the deep end with the World War II analogies. No doubt some of the woo-friendly commenters here (and perhaps even some of those on “our” side) will react to this post with the conclusion that Gorski has completely lost his friggin’ mind–or at least lost all sense of proportion and restraint. (I can see Steve holding his head in his hands, muttering under his breath something about what on earth was he thinking when he invited me as a regular on SBM.) But desperate times call for desperate rhetoric. Just as I painted an intentionally bleak picture in my last post as a wakeup call, in this post, I want to do the same thing, but without being so dark. I want you to realize that, as bad as things are now, there is still some reason to hope, but nurturing that hope from a sputtering spark to a roaring flame will require fuel, and that fuel is passion coupled with an understanding not only of what we’re up against but the will to act against it. Either that, or I’m bipolar, with my original post being written when I was in a depressive phase and this post being written while I am in a manic phase. You be the judge.
Fortunately, I also don’t think that resistance to the infiltration will require that a “really futile and stupid gesture be done on somebody’s part,” as Animal House tells us in the scene above. Taking the long view, I realize that it may, however, require the proverbial fight with conventional weapons that will take years. It will, mainly, require that advocates of science-based medicine who have until now been keeping their head down become involved and vocal. Too, it will require years, if not decades, of resistance, diplomacy, and, yes, public relations, as much as we hate that term. It will require fighting smart, because that’s the only way to win against forces that are so much stronger than us, and it may require us to get our hands dirty in politics. It may require strategic retreats, and it will definitely require that we as advocates of science-based medicine be willing and sufficiently open-minded to embrace any aspect of CAM that is actually scientifically validated, thereby making its strength our strength. Dedication to science and patient care demands no less, because, as I have said many times before, there is no such thing as “alternative medicine,” just medicine. Most importantly, however, it will require that we rethink what we are doing, because clearly what we are doing is not working. A major change in strategy is clearly required.
Whatever this battle requires, though, I ask myself and I ask the readers of this blog who are dismayed at the infiltration of woo in medicine: “Are we just the guys to do it?”
Well, are we?
And, if we are “just the guys (and gals) to do it,” then what, specifically, are we going to do? We’re getting our posteriors handed to us in a sling right now. Something has to change. I don’t yet know just what yet (stay tuned), but, as Pete Townsend once so eloquently put it, “The music must change.”
My answer to the question “Is resistance futile?” is this: Hell, no! And, even if it were, when the day comes when I lie on my deathbed looking back at my life, I would never be able to forgive myself if I just gave up now. But we need to realize something. If we don’t change tactics, we will lose, and it could be many decades before scientific medicine recovers.
So I ask again: What are we going to do about it?

SBM in 2018 and Beyond
A look back at SBM in 2018 and the future of SBM.
Steven Novella on December 19, 2018
Link

This is my last post for the year (although I may squeeze one in during the holidays, but no promises), and I often like to take the opportunity to take a look back at how SBM is doing.
This marks the end of our 11th year publishing SBM, and this article will be the 3,285th we publish. I like to think we achieved our primary goal of providing a respected and reliable source for health information, for both professionals and consumers. We focus on bad science in medicine, but also on improving the quality of medical science in general. I personally have learned a great deal over the last decade, exploring the nuances of how to best use scientific knowledge to improve medical practice.
The last decade has also been a time when medical science has been under assault. Pseudoscience and health fraud has infiltrated our schools, our journals, pharmacies, hospitals, and clinics. Unfortunately there are billions of dollars to be made selling fake medicine, and essentially none in warning of the perils of pseudoscience in medicine. But despite the fact that we are essentially a small group of volunteers, we have a disproportionate impact on the conversation. We have been well-reviewed and are highly referenced. We have been invited to the NIH, and have been quoted directly in our recommendations to the FDA and FTC. We have regular media contacts, and are basically an important part of the conversation.
We had another great SBM conference connected to the NECSS conference in New York, and plan to participate again next year.
We have also successfully fended off a harassing law suit. I won my case in summary judgement, won an anti-SLAPP part of the suit, was awarded partial fees, and all this was upheld in appeal. We are still, amazingly, at the tail end of this saga, but I just learned that the court approved our last request for fees. 2018 may see a final end to this, and if not then early 2019.
However, at about the same time we learned of a new lawsuit, this one targets all the editors of SBM. This one, in my opinion, is also frivolous, harassing, and a direct assault on our first amendment rights, and our duties as professionals. I suspect it won’t go anywhere, but even a frivolous and bogus lawsuit takes incredible time and money to deal with. We will keep you updated.
That is the nature of what we do. If you target charlatans and con artists, you are going to get backlash from people who don’t exactly respect truth, professionalism, and following a valid process. People who sell fake medicine also tend to be flush with cash, which gives them the ability to use their funds for legal thuggery. We will endure, and we will tenaciously fight every such attempt at silencing our valid opinions, and we will win – because our only interest is in providing factual information and thoughtful analysis.
This, by the way, is also why we need to continue to lobby for effective anti-SLAPP laws, in every state and at the federal level. It should not be financially ruinous to defend your first amendment rights – otherwise, you don’t actually have that right. It exists only at the whim of those with more money than you.
SBM also continues to have ambitious plans for the future. This year we started a Patreon page and are already getting incredible support. We appreciate everyone who supports SBM, and we are almost at the point of self-sufficiency. Beyond that we have also laid out our goals, that will scale with our support. We plan to start an SBM podcast, make videos, hire full-time staff, and eventually finance our legal-defense fund.
We also have plans to combine SBM and the Society for SBM into one organization. Right now SBM is a project of the New England Skeptical Society (NESS) and the SfSBM is a separate entity. It makes sense to just combine them into one non-profit and then try to grow that organization.
We continue to recruit new writers and contributors. We receive many submissions, which is great, but also could use more regular contributors. There is also so much more that we can do as we grow. We are fighting on dozens of fronts – against anti-vaccine propaganda, the normalization of health fraud, the watering down of necessary regulations and licensing of quackery, and attempts to change the rules of science to accommodate pseudoscience. Most of us also have day jobs.
We have developed a vibrant community, and continue to grow that community (the recent addition of a Discord SBM server helps) but we need to grow further, to organize more effectively, and to marshal those who care about science-based medicine. We do this because we care about our patients and our society, and we can see how important it is to base medical practice and regulations on valid science. Not everyone shares our interest, or they are simply confused about how science works. Most academics are simply unaware that their profession is being hollowed out by pseudoscience. Most regulators are not scientifically literate.
So we are engaged in a massive up-hill battle, but we are dedicated. We appreciate all of those who support our work and have our backs, and we can grow as that support grows.
Thanks everyone for another year of science-based medicine.

Science-Based Medicine in the New Year
As 2018 ends, the managing editor of Science-Based Medicine comments on the future of SBM.
Davie Gorski on December 31, 2018
Link

Today is New Years Eve, the last day of 2018, thus giving me the honor of producing the last post of the year for Science-Based Medicine (SBM). I therefore thought that I’d take a little time to discuss my plans as editor for SBM in 2019. Given that our fearless leader Steve Novella has already discussed things that have happened in 2018, I’m not going to dwell much on that, other than maybe one thing. Because I’m still technically on vacation for the holidays, this post will be shorter than usual as well, probably a lot shorter. After all, who wants to read one of my 3,000-5,000 word epics on New Years Eve?
Before I discuss the future, it is worthwhile to briefly mention three key developments that occurred in 2018. First, we started a Patreon page to support our work financially, thus finally (we hope) freeing us from the tyranny of automated ad services, which sometimes embarrassed us by serving up ads for quackery based on the keywords in our posts. Who knew that the algorithms used by Google and other ad services can’t distinguish between keywords in articles promoting something versus criticizing something as pseudoscience? I kid, of course. Unfortunately, no ad service that we tried provided adequate controls to keep the quack ads from popping up from time to time. Second, as Steve mentioned before, it is true that several editors of SBM (myself included), both current and former, were named in a libel suit in November. I won’t say more than that on the advice of my lawyer other than that we are, of course, going to fight it in court. Finally, we are planning on merging SBM and the Society for Science-Based Medicine, because it will allow us to combine all our efforts under one “brand.”
So what about 2019 and beyond?
As managing editor, I’ve been blessed with an incredibly strong stable of writers and other editors, such that I really don’t have to do much, if anything, to guide them. Sometimes I’ll suggest a topic (usually in response to requests from readers). (I also thank our regular proofreader, because he makes sure that I don’t have to do much editing of the copy.) Occasionally, I’ll point out errors in a post (usually minor) or that I disagree with the major point being made in a post (rarely), but for the most part I can just let our regulars churn out posts about whatever strikes their fancy on a given week.
We’ve also had remarkable stability in that the average length of time that our current regulars have been blogging for SBM is nearly nine years, with the “most junior” having been with us for “only” six years. Unfortunately, longtime blogger Mark Crislip left us in 2017, and here’s where the key thing I need to do comes in. We still haven’t found a regular replacement for him; so every other Friday there is either a guest post (if we have any good ones approved and ready) or there is no post. Stability has been good, and our current lineup is stellar, but this failure to find a regular replacement for Mark reminds me that we don’t have a deep bench. Actually, we don’t have a bench at all. I’m as far away from my goal of having at least one post every day, weekends included, as I’ve ever been—farther, actually. (I should have recognized this problem more acutely a couple of years ago when Harriet was off for several weeks with injuries suffered from a fall.) None of us is getting any younger, and perhaps our lineup has been too stable for too long. There are advantages (obviously) to having had such a stable lineup of excellent bloggers for so long, but fresh talent could reinvigorate the blog.
So the most pressing need SBM will have in 2019 is the same one it had in 2018: Finding more regular bloggers. New regulars are most likely to be found as result of the part of my job where I actually do have to do some actual editorial work. I’m referring to guest posts, of course. We actually do get a fair number of guest post submissions, but unfortunately the vast majority of them are utterly unsuitable for SBM. Many are medical clickbait, general medical articles that would be suitable for a general medical blog but not SBM, listicles, and the like. These inquiries I usually ignore, although if someone is particularly persistent I might respond and tell him why his post is not suitable for SBM. Sometimes we get submissions from someone trying to beat his or her favorite hobby horse that’s not particularly relevant to SBM. Hilariously, some are from proponents of alternative medicine touting the benefits of supplements, modalities like reiki or homeopathy, or various other types of quackery. I always laugh and respond to these with a comment along the lines of, “Obviously, you’ve never read our blog. You really should before submitting.” Then, every so often, there’s a diamond in the rough, someone who can write and “groks” what we are about at SBM. These submissions are uncommon enough, but even rarer are submissions from someone who not only “groks” what we are about and can write well but also could contribute a regular post every week or every other week; i.e., could become a regular.
That brings us to what I’m looking for in a new regular. I’ve long wanted to bolster SBM in areas where we are weaker in expertise. From my perspective, we could really use experts and specialists in (but not limited to) these areas: mental health, OB/GYN and women’s health, epidemiology, public health, cardiovascular disease, surgical subspecialties, immunology, and basic biomedical sciences. (Feel free to leave suggestions of other areas where SBM could use more expertise in order to cover properly and potential other bloggers.) None of this means I won’t consider talented bloggers who “grok” SBM in specialities that are already represented among our bloggers, but I will tend to prioritize filling holes in our medical coverage portfolio wherever possible.
I also realize that beggars can’t always be choosers. Let’s face it. SBM is an unpaid gig, albeit with a large readership for a medical blog and strong Google juice. It’s a well-established brand, so to speak. Being associated with SBM can thus help a blogger get noticed, leading to speaking gigs, either at NECSS or other skeptical conferences, and to reporters contacting him for quotes in their stories. Unfortunately, though, the single biggest hurdle to clear in beefing up our lineup is that most good bloggers who could be a “fit” as part of SBM tend to have their own blogs and/or other writing gigs, some even in the mainstream media. Of course, we only ask for one week of exclusivity, after which a blogger can post on his own blog, and our bloggers retain the copyright to their work; they only give us permission to post it, which they can rescind at any time. Also, as you know, some of our current regulars have blogs of their own, and Steve and Harriet are frequent contributors to skeptical magazines.
Also, in the new year, I plan on trying to remedy my biggest deficiency as editor, namely the speed with which I make decisions on whether to accept or reject a guest post. As described in our Submission Guidelines, guest posts that make it past my initial screen (i.e., that aren’t clickbait, listicles, general medical posts, or posts promoting quackery but actual posts saying something interesting related to SBM) go through an informal “rough and ready” peer review in which I solicit comments from other editors, usually without saying anything about my opinion of the submission, so as not to bias anyone. Unfortunately, that can drag on for a long time, and I’m not always on top of making sure the reviews come through. That’s why I also apologize for times when our determination regarding whether a guest submission is suitable for SBM takes too long. My only excuse is that those of us evaluating them have demanding jobs. For example, as the time gets closer to a major NIH grant deadline, my processing of guest submissions often grinds to a halt, or at the very least it slows way down. (As a heads up, I’ll mention that the next big NIH deadline is February 5, and, yes, I will be submitting at least one R01 then.) In any event, if I have a New Years resolution it’s to do better. The last thing I want to do is to scare away a potential new regular by unintentionally stringing them along.
Also, our presence on social media has been spotty, with our Facebook page and Twitter feed not always being updated regularly. I’d like to try to do something about that as well, even though that’s not part of my purview as managing editor.
So, to summarize, my main goals for SBM for 2019 are:
     1.  to be faster and more reliable and responsive evaluating and publishing guest posts and to try not to leave bloggers hanging as long, in order to get more repeat submissions from promising bloggers who might be able to become regulars;
     2.  to find enough additional regulars to be able to post seven days a week (although I’ll settle for five or six given how difficult this will be);
     3.  to fill as many gaps in our expertise as I can; and
     4.  to get our social media up and running better (if I can).
As always, the goals of SBM remain unchanged:
    • To champion science as the basis for producing the most effective and safe medical care.
    • To identify and write about areas where medicine is not based on the best science.
    • To combat the continuing infiltration of pseudoscience and quackery into medicine and public health policy.
    • To educate.
Finally, trials and tribulations aside, I thank you all for your continued readership and support. I also thank Steve for founding SBM and inviting me to be a part of it from the beginning, as well as our editors and all of our contributors for all their hard work and great blogging. I also thank our copy editors/proofreaders, past and especially present, for fixing typos and helping get guest posts ready to publish. We certainly challenge them with our many typos. (Cue the Mark Crislip jokes.)
SBM started on January 1, 2008, which means that we’re entering our twelfth year, hard as it is to believe. I certainly plan that twelve years from now we’ll still be going strong. If you want to be a part of what I envision as an even more successful future for SBM, go ahead, give it a try. Submit a guest post or, if you’re an established blogger, throw your name in the hat for a regular spot on our blogging schedule. If you’re not a blogger, suggest bloggers or areas where we need to bolster our expertise, or, failing that (mandatory begging commences), support us on Patreon to help keep us going.

   Critical Thinking: Basic Principles

Science-based medicine versus other ways of knowing
It has been our position that science is the most effective means of determining medical treatments that work and whose benefits outweigh their risks. Those who promote pseudoscientific or prescientific medicine, however, frequently appeal to other ways of knowing, often ancient knowledge from other cultures and pointing out deficiencies in SBM to justify promoting their treatments. Do their justifications hold water?
David Gorski on June 11, 2018
Link

The one principle guiding Science-Based Medicine (SBM) from the beginning was set forth by our fearless leader in the very first post on this blog over ten years ago:
The philosophy of this blog, at its core, is simple: Safe and effective health care is critical to everyone’s quality of life; so much so that it is generally considered a basic human right. The best method for determining which interventions and health products are safe and effective is, without question, good science. Therefore it is in everyone’s best interest for health care to be systematically evaluated by the best science available.
We strive every day to continue to make the above statements the philosophy of this blog. Contrary to what some critics claim, we do not exempt “conventional” medicine from our skepticism, although we do focus more on pseudoscience and quackery that has been increasingly finding its way into academic and community medicine, mainly because it is a niche that needs to be filled. Unfortunately, promoting rigorous science remains as difficult now, if not more so, as it was in 2008. Scientifically unfounded health scares, antivaccine views, anti-GMO activism, and the increasing respectability of quackery like acupuncture, energy medicine, and the like leave SBM under assault as never before.
Other ways of knowing about medicine
Because so much of what is being “integrated” into medicine as “complementary and alternative medicine” (CAM), “integrative medicine,” “integrative health,” or whatever the next nom du jour will be is either unproven, a distortion of existing SBM, or outright quackery, proponents of integrating pseudoscience into medicine—of course, that is not how they would characterize themselves—can’t win on the science, at least not when it is accurately represented. So instead they use other techniques, and one of the most common tropes to which they resort is the appeal to other ways of knowing. As they say in law, if you have the facts on your side, hammer the facts. If you have the law on your side, hammer the law. If you have neither the facts nor the law, hammer the table. It’s a little different in medicine, where, if you have science on your side, you hammer the science, but if you don’t you hammer anecdotes, sow doubt, or appeal to other ways of knowing.
Regular readers will be familiar with this particular tactic. We’ve seen it used by all manner of quacks, cranks, and charlatans, ranging from homeopaths to Dr. Oz to Deepak Chopra. Basically, when science doesn’t support your argument, an appeal to other ways of knowing takes the form of dismissing the science that refutes your argument by saying something along the lines of “Science doesn’t know everything” (which is, of course, true, but irrelevant) and then appealing to a different “way of knowing” that doesn’t involve rigorous science. The appeal to other ways of knowing is basically a form of special pleading, in which it is claimed that your belief should be exempt from rigorous scientific evaluation.
Unfortunately, there is one form of this special pleading that is gaining some currency, and that’s basically carving out an exception to the requirement for rigorous science for “traditional medicine,” be it Ayurveda, traditional Chinese medicine (TCM), aboriginal medicine, or folk medicine practiced around the world. A while back, I discussed how the World Health Organization (WHO) has included a section of ICD (International Classification of Diseases) codes in the new ICD-11 system for TCM codes. Some of these codes included diagnoses like the bladder meridian pattern, triple energizer meridian pattern, liver qi stagnation pattern, and other “diagnoses” derived from the prescientific vitalistic ideas upon which TCM is based. Of note, ICD-10 (the current version) is the standard that the whole world uses to classify diseases. Government health agencies use it, as do private insurers to determine reimbursement for services. The TCM section (International Classification of Traditional Medicine, or ICTM) in ICD-11 will be optional at first, but I predict that it won’t be too long before it becomes mandatory. In a similar vein, the Chinese government recently passed a law designed to promote the use of TCM and facilitate the export of TCM remedies, in effect lowering the scientific bar for ideology and profit.
Last week, Steve Novella took note of this phenomenon, which was referred to as “Indigenous ways of knowing” (IWK), taking note of a recent article by Josh Dehaas, which is highly critical. I had come across the article as well, but I had also come across a couple of others, one by former MD turned philosopher Paul Biegler (which was a lot more wishy-washy than Dehaas’ article bordering on being an apologia for indigenous ways of knowing), and an article in a CAM journal advocating different standards for regulating TCM and other traditional medicines.
Colonialism as a justification for accepting other ways of knowing about medicine?
Of course, as both articles point out, there is history here. Many of the cultures whose medicine is being promoted have a long history of having suffered predation and oppression under the colonialism of European and American powers. This cannot be denied and is a powerful force making claims for IWK sound more compelling. How? Well, let’s look at what Nadine Ijaz and Heather Boon write in justifying differing standards for regulating TCM. I’ll note that we’ve met Heather Boon before. She’s the Dean for the Leslie Dan Faculty of Pharmacy, University of Toronto, where she’s supported a clinical trial of homeopathy, mischaracterized Steve’s and my position with respect to clinical trials of CAM, and generally shown herself to be a true believer in unscientific alternative medicine. In any event, Ijaz and Boon write:
An important issue that frequently goes unaddressed in scholarly discussions of T&CM [traditional and complementary medicine] pertains to the historical and ongoing impacts of European colonization on traditional medicine systems and practices across the globe. As documented and discussed elsewhere,6–9 traditional medicine treatments and practices have long been subjugated, devalued, co-opted, and in some cases decimated across the globe within the context of European colonization. Still today, many indigenous healthcare systems remain under threat due to colonization’s impacts.10
Biomedicine’s globalized dominance, as Hollenberg and Muzzin have elaborated, is far less the result of biomedical science’s evidenced efficacy than it is a feature of the ongoing sociopolitical subordination of precolonial indigenous knowledge systems and related healthcare practices.11 Traditional medicine continues to be in widespread use, and in many jurisdictions (particularly in the global South) represents the “mainstay of healthcare delivery.”1 However, considerable political, research, economic, and institutional capital continues to sustain biomedicine’s pre-eminence in state healthcare systems worldwide.11,12 Regardless, indigenous systems of medical knowledge remain important resources not only within their specific cultural contexts but also as “critical alternative models for resolving health crises on a global scale where biomedical and technological solutions fall increasingly short.”2
The study and reframing of traditional medicine approaches using biomedical conceptual frameworks and language have been used, for many decades, arguably as a strategy to increase their perceived legitimacy within biomedically dominant healthcare systems.12 This has included the increased adoption of biomedical subject areas in the curricula of institutionalized training programs for codified traditional medicine systems such as Chinese medicine, Ayurveda, and Unani, as well as an increasing body of biomedical-style research conducted on particular traditional medicine therapies.
This is the basic narrative of IWK: “Western” biomedical science is a cultural construct whose dominance is due not to its success in diagnosing, treating, and, in some cases, eradicating disease but rather to past imperialism and colonization by European powers. A corollary to the above claim is that attempts by practitioners and proponents of traditional medicine to achieve scientific legitimacy are not because science is better, but because the biomedical model predominates because of the prior subjugation (and, in some cases, the continued oppression) of indigenous peoples. Add to that an appeal to popularity, namely that many millions of people still use traditional medicine as their primary form of health care, and you have an argument that can seem powerful, both to indigenous people using traditional medicines and to some “Western” scientists or physicians—like Heather Boon.
Indeed, Boon goes even further. First, she points out that nearly a quarter of all modern medicines are derived from natural products, many of which were first used in traditional medicine. While I suspect she exaggerates a bit regarding just how many of these natural products were discovered by traditional practitioners, it is true that some were. Of course, the story is not always that simple, as when TCM advocates claimed that the awarding of the Nobel Prize in medicine to Youyou Tu for her discovery of the anti-malaria compound Artemisinin, which was used in TCM. It turns out that considerable “biomedical science” had to go into isolating and validating the active component. This is the sort of thing that Ijaz and Boon complain about, namely how most traditional herbal medicines have undergone “considerable recomposition en route to pharmaceutical usage,” which they describe as a process that “privileges biomedical epistemology while erasing/negating the remedies’ indigenous cultural origins and epistemic underpinnings.”
See what I mean? By this appeal to IWK, “Western” scientists are damned if they do, damned if they don’t. Science is portrayed as being just another way of knowing whose preeminence in medicine is due more to cultural and national hegemony than to actual effectiveness, and if that “Western” science actually tries to use traditional medicines its application of science to testing and formulating them into safe and reliable forms ends up “erasing” the cultural origins and epistemological basis for their use, or, as Ijaz and Boon characterize one problem with regulating traditional medicine, the “historical circumstances (and resulting evidentiary tensions) that surround traditional medicine’s political subjugation to Western biomedical knowledge systems.”
Biegler, unfortunately, appears to be sympathetic to, if not entirely buying, these sorts of arguments:
Few dispute that traditional cultures should be protected and knowledge preserved. But that is a long way from saying that cultural longevity confers legitimacy on a health treatment. By turning the torch on colonialism are the authors sidestepping the awkward fact that the real threat to traditional medicine comes from science, a discipline that bridges the global North and South?
The back-story is that practitioners of traditional medicine (Ijaz is a medical herbalist and shiatsu therapist) have good reason to see the randomised clinical trial (RCT) as a threat. One criterion of the US Food and Drug Administration (FDA) for approving medicines is that they been shown superior to placebo on two RCTs. It’s a standard that could ring the death knell on some TCM practices, should they be compelled to conform to it.
Which, of course, plays to the authors’ point that the Western model threatens to extinguish many venerable and ancient therapies.
Remember, though, that plenty of Western medicines fall at the very same hurdle. An infamous recent example was researcher Irving Kirsch’s use of Freedom of Information to unearth 47 failed antidepressant trials from the FDA, trials subsequently buried by the parent pharmaceutical companies.
If Western medicine is predatory, then, it also eats its own.
Of course, I can’t help but note that more recent evidence clearly supports the efficacy of antidepressants, although the effect size appears to be smaller than previously thought. I also can’t help but note that, in general, science is self-correcting, while fully conceding that the process of self-correction is often quite messy and takes longer than we wish it would. All those traditional medicines have been used for centuries or millennia based on anecdotal evidence passed down through the generations and are basically never abandoned as ineffective. That’s a big difference. “Western medicine” is indeed predatory, and it does eat its own when later evidence shows a treatment to be ineffective or carry more risk than the severity of the condition being treated warrants taking.
“Subjective” knowledge
Dehaas describes quite well what IWK really means, illustrating it by quoting a professor at UT who teaches a course on IWK, which is, alarmingly, being increasingly infused into the curricula of universities in Canada:
In an introductory IWK lecture, Paul Restoule, a professor at the University of Toronto’s Ontario Institute for Studies in Education (OISE), starts off by telling his class that “knowledges” are subjective. He also claims that the mere act of defining IWK is “problematic,” since any definitions would use “Western knowledge” as a frame of reference. This is not unusual. I’ve been writing about higher education for a decade, and have investigated the issue of IWK on different campuses. Invariably, my attempts to determine the exact parameters of IWK always meet with this somewhat gauzy, defensive response. Oddly, the most zealous proponents of IWK also are the ones who are the most reluctant to describe what it is.
Restoule claims that, for Indigenous people, “the senses can know more deeply and concretely than knowledge gained through reading and being told.” He asserts that “knowledge is sometimes revealed through dreams, visions and intuitions.” And he offers a Venn diagram with a circle for “Western science”—“limited to evidence and explanation within [the] physical world” and “skepticism,”—overlapping somewhat with a circle for “Indigenous knowledge,” which is described as “holistic,” involving a “metaphysical world linked to moral code” and “trust for inherited wisdom.”
This is, of course, correct. Basically, what IWK, at least with respect to traditional medicines, appears to boil down to is to trust that one’s forbearers got it right all those centuries or millennia ago. It is also based on prescientific, mystical, and/or religious beliefs, as Frances Widdowson, a professor of political science at Mount Royal University in Calgary, is quoted as saying:
Whenever proponents are asked to define IWK, “at some point in the conversation, postmodern relativism begins to enter into it,” she says. When asked to explain the unique “ways of knowing” exhibited by Indigenous peoples, advocates tend to describe either folk knowledge or spiritual beliefs, she adds. These may indeed be described as “alternative” ways of knowing. But their alternative character originates in the fact that they present themselves as exempt from the expectation of rigorous scrutiny that typically is applied to claims made by academics.
Precisely. “Western” scientists who promote IWK are, in essence, saying that we should trust “ancient knowledge” and that such ancient knowledge is just “another way of knowing” that is no worse or better than science, just different, even if the knowledge is based on tradition, religion, and prescientific understanding of how the body functions. I should, however, point out that I prefer the term “other ways of knowing” over “indigenous ways of knowing,” given that it is a more general term that encompasses much more of the special pleading that CAM apologists indulge in to claim their treatments are equivalent to science-based medicine without being so specific to a culture or potentially loaded with the baggage that comes with the long and violent history of colonialism and imperialism. Also, I can’t help but note that two of the most popular traditional medicines, TCM and Ayurveda, came from highly advanced civilizations, which is not captured in a term like “IWK.”
There’s also more than a little bit of a racist tendency that has been called the “cult of the noble savage,” wherein indigenous peoples are romanticized as being “purer,” more in tune with nature. Uncorrupted by modern civilization, the noble savage is good and can possess hidden knowledge that we “civilized” (and therefore corrupted) people do not. It’s a not uncommon trope in literature and the arts. (Think the Lakota tribe in Dances With Wolves or, in a science fiction context, the The Na’vi in Avatar.) Not coincidentally, it’s also a variant of the story of the fall of man in Genesis. Indeed, if you want to know why I use scare quotes when referring to “Western” medicine or science, it’s because dividing science into “Western” (cold, rational, reductionist) as compared to “Eastern” or cultures other than European (which includes nations like the US that started out as European colonies) as more “naturalistic,” mystical, and in tune with nature and ancient knowledge is just a variant of the racist noble savage myth.
Sadly, it’s also an attitude that, thanks to postmodern-like relativism, often infects CAM proponents.
Worse, accepting such IWK with respect to medicine can result in real harm:
Widdowson recounts the story of an Inuit man in northern Quebec who got frostbite so severe that his boots froze to his feet. Instead of going to a doctor or warming up his feet, he turned to an elder, who suggested he pack them in wet snow. Eventually, he was coaxed by the RCMP to a hospital where doctors informed him that his reliance on traditional treatment methods might have cost him his feet.
I also can’t help but recount the case of a First Nations girl in Ontario who suffered from leukemia whose mother wanted to choose “aboriginal medicine” (although in reality what she was choosing was the quackery of a white con man who preyed upon Canadian aboriginal people). The results were tragic. It doesn’t even have to be a different race, either. I’ve recounted the story of an Amish girl with cancer whose parents decided to abandon SBM and treat her with traditional Amish medicines. The concept is the same. The “simple” people know things that we modern people do not.
Distrust of “Western science” and the role of the shaman-healer
Of course, it’s understandable why many indigenous people are distrustful of their former colonial rulers. As I discussed when considering the case of the First Nations child with cancer, Canada has a horrible history of placing children of indigenous people in residential schools, whose express purpose was to remove the children from their own culture and assimilate them into Canadian culture. There, many aboriginal children suffered physical and sexual abuse, and it is estimated that 6,000 died over the 100+ year history of the schools. It’s not surprising that there still exists in Canadian aboriginal communities a great deal of distrust of the government and the medical system that facilitated such cultural imperialism, the same as the native peoples in the US very much distrust the government that drove them off their ancestral lands and onto reservations and as other indigenous peoples distrust their former (and, all too often, continuing) oppressors. It’s very easy for them to view “Western” science as just another tool of oppression that delegitimizes their traditional medicine.
But why do “Western” scientists buy into it? I think it’s far too flippant of Dehaas to say that “most of those who sign on are simply afraid of being called racists,” although there could well be element of that. Another motivation for embracing other ways of knowing related to traditional medicine almost certainly includes an element of a longing to be the shaman-healer, something that no less a personage in the world of CAM than Dr. Mehmet Oz himself has expressed in an interview:
“I would take us all back a thousand years, when our ancestors lived in small villages and there was always a healer in that village—and his job wasn’t to give you heart surgery or medication but to help find a safe place for conversation.”
Oz went on, “Western medicine has a firm belief that studying human beings is like studying bacteria in petri dishes. Doctors do not want questions from their patients; it’s easier to tell them what to do than to listen to what they say. But people are on a serpentine path through life, and that is the way it is supposed to be. All I am trying to do is put a couple of road signs out there. I sit on that set every day, and that is what I am focussing on. The road signs.”
It’s that role, that some doctors crave (and, to some extent, understandably so), that of the healer. It’s a large part of why most doctors went into medicine in the first place. Why? I suspect it’s part of reclaiming something that many perceive as having been lost over the last couple of decades, the physician-patient interaction. As the financial pressures of practicing medicine have grown and patient face time has declined, it’s understandable that some physicians would like to reclaim “the way it was,” whether it ever really was that way or not. They yearn for the days when doctors were “healers” and shamans, the way medicine was for hundreds and hundreds of years before science intruded. Indeed, I’ve lost track of how many times I’ve seen rhetoric from practitioners of “integrative” medicine that basically says just this, usually with the implication that to attain that hallowed role of healer requires the embrace of various prescientific treatments.
Indeed, there still exists in the collective consciousness a concept of the physician along the lines of Dr. Marcus Welby and Dr. James Kildare, kindly, benign figures whose influence and good intent were unquestioned by patients, doctors who in essence functioned as modern day shaman-healers. Doctors also long for a perceived time when the will of the physician was generally unquestioned, and patients did what they were told. Of course it’s all romanticized. Certainly shamans were respected and could have empathetic relations with those whom they treated, but they couldn’t really do much else, given the limited their ability to actually treat serious disease and injury was.
I also can’t help but point out that embracing other ways of knowing about medicine has become a business opportunity:
We also should remember that, however warm and fuzzy the principles of IWK may sound, it also has become a business opportunity. In recent years, the demand for more IWK in curricula has created a niche for those who present themselves as an expert in this vaguely defined area. As with other efforts to expand the influence of other cultures in schools and businesses, IWK draws in educators, consultants and administrators whose job is to help these institutions match action to words. These programs also are sometimes accompanied by demands that those who teach the subject be allowed to do so without the normally required credentials, as is recommended in a recent report prepared for Ryerson University.
Also, in the case of TCM, appealing to other ways of knowing has become an even bigger business, promoted and protected by the power of the Chinese government. Indeed, China’s TCM industry grosses $116 billion a year and represents nearly 30% of China’s entire pharmaceutical industry.
Science, culture, and other ways of knowing
Unlike Steve, I’m not going to say that science necessarily transcends culture, at least not today. It can, however, transcend culture, and Steve is correct that a goal of science is to develop a way of knowing and methods that can work for everyone everywhere, regardless of culture, to investigate how nature works and use that knowledge for the betterment of humankind. There is also no doubt that, in medicine, science has produced the goods, saving more lives and preventing more deaths than any “way of knowing” that came before. No one, least of all I, should claim that science is perfect. Obviously it’s not. Advocates of applying other ways of knowing to medicine (like Boon) might love to trot out examples of SBM’s failures or imperfections as a justification for embracing prescientific methods of treatment, but until the traditional medicine they embrace can do what SBM has done and still does, pointing out SBM’s imperfections does not justify abandoning science and retreating to the past. Yet, that’s what advocates of traditional medicine ask us to do.
None of this is to say that indigenous knowledge should be ignored. As scholars have pointed out, the local knowledge of indigenous peoples can be valuable in studying climate and ecological change. The example of the Nobel Prize for Artemisinin shows that some traditional medicines can be turned into highly effective medicines, although it’s hard not to note that the experience of screening natural products for medicinal properties (i.e. pharmacognosy) suggests a low wheat-to-chaff ratio. Respect for cultural knowledge, however, does not justify exempting ancient claims about medicine from scientific scrutiny or using a lower bar of evidence to use and regulate traditional medicines than we do for SBM.

A Culture of Standards Matters
Perhaps the most dangerous effect of the alternative medicine movement has been an erosion of the culture of dedication to science and standards within medicine. This has to change.
Steven Novella on September 12, 2018
Link
We are often asked “What’s the harm?” when it comes to apparently benign placebo interventions for subjective symptoms. “If people feel better, who cares if it’s pseudoscience?” We have had to point out, endlessly, that there is a great deal of potential harm – delay or even avoidance of effective treatment, opportunity costs, financial harm, psychological harm, and overall promotion of unscientific medicine.
Perhaps the most insidious form of harm, however, comes from the erosion of standards within the medical profession. Medicine is only science-based and ethical if its practitioners are science-based and ethical. There is a standard of care within medicine to maintain a minimum of both quality and ethics.
The erosion of standards takes two basic forms – weakening of the system that maintains standards, and a softening of the culture of dedication to patients, science, and ethics among practitioners. I have discussed before how proponents of alternative medicine have been (successfully) seeking to weaken the regulations that maintain the standard of care, to remove protections for consumers, to change the rules of science to favor their preferred treatments, and to introduce magical thinking into medicine. They are literally changing the laws and regulations to lower the guardrails.
But perhaps an even more damaging effect of the alternative medicine movement is the slow change in the culture. You can post whatever speed limits you want, but if the culture is such that drivers routinely ignore the speed limits, and police routinely look the other way (or simply don’t have the resources or authority to do much about it), the posted limits become meaningless.
Regardless of your political leanings or whether you think the current administration is a good or bad thing for our country, it has clearly demonstrated that our elaborate system of checks and balances mean nothing if individual politicians are not motivated to adhere to them. No system and no institution is robust enough to withstand a culture of disregard for established standards.
This is exactly what I am seeing among a disturbing number of my medical colleagues. Alternative medicine propaganda has been slowly leaching into the medical culture, it has infiltrated medical education and is therefore changing with the generations, and it has weakened our collective dedication to a rigorous science-based standard of care.
I now see many more instances of otherwise reasonable science-based practitioners routinely referring to chiropractors, naturopaths, or acupuncturists without really knowing the kind of care their patients will be receiving, for example.
Perhaps the most common example is medical practitioners actually recommending pseudoscientific interventions, mostly for symptomatic treatment. They have bought into the “What’s the harm?” argument, or the placebo medicine argument of “Who cares if it really works if the patient subjectively feels better?”
This has always been a tempting practice. Everyone will accumulate patients who are difficult to treat, or have symptoms that are refractory to standard interventions. It is tempting to essentially punt, give them some placebo intervention to distract them (for a while, at least) so at least they will call your office less often. It is easy to convince yourself that this is OK – it’s OK to recommend pseudoscience to your patients, as long as its “harmless” pseudoscience.
The harder road, but the one that is better for your patients, is to carefully vet any practitioner or any treatment that you recommend, to make sure they are above the waterline in terms of scientific legitimacy and ethical standards. At the very least, the ethical standard of informed consent demands that you give your patients adequate information.
When patients ask me about an unscientific intervention, I tell them, “I don’t recommend that because I have reviewed the evidence and it is: not convincing, not plausible, too preliminary, or may even demonstrate lack of efficacy.” If they still decide to pursue it, at least you gave them your honest, professional, and informed opinion, and you did your due diligence.
Even worse is the increase in the number of practitioners and institutions who are increasing their income by directly selling pseudoscience. I know many local practitioners who are regular doctors, but they also have a cryotherapy chamber in their office, or sell aromatherapy supplements. Hospitals now routinely offer alternative treatments as add-ons for “quality of life”. Sure, we’ll have someone wave their hands over you for a fee. It’s almost free money.
The only thing that is really keeping doctors and hospitals from practicing rank but lucrative quackery is a culture of dedication to scientific standards in medicine. As that culture erodes, quackery increases.
And that is what the alternative medicine movement has always been about – it is about promoting profitable snake oil. Pesky laws, regulations, and the standard of care have been an obstacle to selling snake oil, and so the purveyors of all sorts of dubious treatments have campaigned to weaken the laws and regulations, and to abolish the standard of care.
They have largely succeeded. I don’t mean that there isn’t dedication to science within medicine, or many practitioners who do maintain high standards. Rather, I mean that the system now happily accommodates anyone who wants to sell snake oil alongside legitimate medicine. It’s all good.
I don’t know if it’s too late to turn this ship around. I’m hoping it isn’t. It will take scientists, academics, and serious politicians to wake up and see what is happening. There are signs of progress here and there, like the retreat of homeopathy over the last decade. It is still possible to make an argument that magic water doesn’t work and should not be accepted as a legitimate part of medicine.
But it will take those of us who clearly see the need for standards and the harm of pseudoscience in medicine to continue to forcefully make our case. Science matters. Standards matter. And while institutions and regulations matter also – the most important thing is a culture of respect for science and standards.

Is defending science-based medicine worth it?
Pseudoscience, denialism, fake news, and disinformation about health are a bigger problem than ever, thanks to social media. As doctors and scientists join lay defenders of science on social media, will they be willing to pay the price in terms of harassment? Or will they decide it’s not worth the hassle? And what about our fellow docs who think that it’s beneath them to debunk quackery, that it is so easy as to be not worth their effort?
David Gorski on January 20, 2020
Link

There are perils involved in communicating science and medicine, in particular in standing up to pseudoscience, quackery, and the antivaccine movement. I’ve discussed some of them before, most recently when I was targeted with an abusive Michigan Freedom of Information Act request by Gary Null’s lawyer Neal Greenfield a few months ago. (I’ll update that story later in this post.) Before that, there was the time that a “Lyme-literate” doctor sued Jann Bellamy and most of the other editors of this blog for a post he didn’t like. Fortunately, it was dismissed, but not before causing considerable trouble. Then there was the time that Mike Adams published something like 40 libelous posts about me over the course of a few months and claimed to have submitted “complaints” about me to the local FBI office. Before that there was the time antivaxxers, led by a particularly nasty piece of work named Jake Crosby, flooded my university’s board of governors, my medical school dean, my department chair, and my cancer center director with complaints about a nonexistent “conflict of interest”. Those are just some “highlights”. There are many more examples.
The reason I’m “reminiscing” (if you will) about some of these incidents is because over the weekend, I came across an article on the perils and price of standing up for science by Anna McKie entitled “Is standing up for expertise a fool’s errand?” It turns out, unsurprisingly, that it’s not just those of us who defend vaccines against antivaxxers and medicine against quackery or scientists like Michael Mann, who defends climate science against its deniers, who routinely come under these sorts of attacks. It’s any scientist who tries to communicate science to the public that gores the oxen of powerful interests or is the subject of denialist campaigns against it, as vaccine science and climate science are.
McKie starts out with the example of Oliver Bernard, a pharmacist who runs the blog The Pharmafist. He came under attack, oddly enough, for pointing out (as I have on multiple occasions) that high dose vitamin C treatments are not effective.
McKie begins:
“When you want to defend science publicly, you don’t get a lot of support,” concedes the pharmacist and science communicator Olivier Bernard. “Even my colleagues were telling me ‘you are so courageous’ – and I thought: ‘Thank you, but can you do something? That would be more helpful.'”
Bernard had provoked a ferocious backlash from supporters of alternative medicine after he spoke out on his blog against their lobbying of the Quebec government to introduce vitamin C injections as a cancer treatment. His explanation of where popular perceptions about vitamin C differ from what has been scientifically demonstrated – interspersed with humorous cartoons – prompted activists to lobby for him to be fired from his pharmacy job and removed from the professional register. They also tried to get his TV show cancelled and urged supporters not to buy his wife’s book (on an unrelated topic). They even resorted to sending death threats to Bernard and his family.
“When you are passionate about what you do, you try to do good work,” he tells Times Higher Education. “But when you come home and people are threatening you, you do begin to wonder if it’s all worth it.”
Sound familiar? It sure does to me. I don’t have a TV show, but I’ve experienced all of the above except for someone trying to cancel my TV show or attacks on my wife’s book, mainly because I don’t have a TV show and my wife hasn’t written a book. This part sounds familiar as well, but less so:
A posting on Facebook about the hate campaign he was facing also provoked a big reaction – but this time it came from the science community in his support. The petition to introduce the vitamin C treatment was rejected and the Quebec government has created a task force to protect scientists who speak on sensitive topics, as well as an inter-professional advisory committee to support healthcare professionals who speak publicly.
That’s what was missing, Bernard believes. Referring to the initial silence from colleagues, he says: “I’m not trying to shame them; I understand why they would not want to do it. But having support is key.”
It’s true. Even now, in the age of social media where finally tech companies, the government, scientists, and physicians are waking up to how easily disinformation is weaponized on Facebook, Twitter, Instagram, and now Tik Tok, science communication to the public is nowhere near as valued relative to its importance to society. The number of doctors and scientists who do it is a far too small percentage of doctors and scientists, and the percentage of those doctors and scientists who specifically speak out about misinformation and disinformation is a relatively small. The reason is likely because of what happens to people like Bernard when they do. If you’re effective speaking out against pseudoscience, denial of science, and quackery, you will be targeted, and this is nothing new. What is new is how easily you can be mobbed and driven from the discussion, thanks to social media. What’s newer is that scientists, physicians, and lay advocates for good science and medicine are finally starting to push back as a group, and my perception is that there are more of them than ever active on social media. Unfortunately, we still remain hugely outnumbered, outfinanced, and outgunned.
The first attack
The following is a story I recount every few years when relevant; so if you’ve heard it before and aren’t interested in hearing it again, feel free to skip this section, although I will mention aspects of this story that I’ve never recounted before. The date was April 2, 2005. Less than five months earlier, I had started my first blog under a pseudonym that many of you know well, a pseudonym that I still use but that is now among the worst-kept secrets in the medical blogosphere or Twitterverse. That day, I received this email from a man named William O’Neill (and, yes, I have kept his emails all these years):
Sir:
You are the owner of the website http://www.oracknows.blogspot.com. Within the website you make reference to my clincal [sic] practice and myself in a manner that is unfactual, libelous and defamatory.
In the event all materials concerning myself, William P. O’Neill, and my clinical practice, the Canadian Cancer Research Group are not removed by the end of business on Monday April 04, 2005, I will initiate action.
Further note, this is a private correspondence. As such distribution, copy or reproduction is by permission only.
Govern yourself accordingly:
William P. O’NeillCEOCanadian Cancer Research GroupSuite 2299 Fifth AvenueOttawa, Ontario, CanadaK1S 5K41.613.239.0220[email protected][email protected]
c.c.
Mr. Joel Taller, LLBGowlingsOttawa, Ontario.
The weird thing is that this threat was based on a single post in which I linked to a post by Australian skeptic Peter Bowditch, who had frequently criticized the CCRG for promoting cancer quackery. Indeed, Bowditch keeps pages on his website, The GAL Chronicles and The CCRG Correspondence File, dedicated to mocking O’Neill for his email attacks. Let me tell you, perusing these webpages as I wrote this post was a real blast from the past. In brief, Mr. O’Neill formed CCRG in 1998 after he and his wife claimed to have cured their then 12-year-old son Liam of a malignant brain tumor and a recurrence using nonstandard treatments. However, doctors who treated Liam said that the tumor was removed by surgery and the boy received radiation and chemotherapy that cured a subsequent recurrence in the boy’s spine. As Dr. Stephen Barrett (who also received legal threats similar to the one I did) recounted:
In 2001, CCRG’s services cost CN$150/hour for assessment and evaluation (typically $750), $1,500 for “development of therapeutic plan, $10,000 ($5,000 in advance and $500/month for 10 months) for “orthomolecular diagnostic and assays,” and $4,000 per “treatment cycle” for “immunotherapy” (maximum 4 cycles per year). Purchasers were asked to sign a “consent” form agreeing “to not hold liable Canadian Cancer Research Group for act or activity as it relates to the treatment of my cancer.” [5].
For several years, CCRG’s online “interview” form asked whether the prospective client had “general interest” in chemotherapy, radiotherapy, surgery, immunotherapy, vaccine therapy and/or homeopathy and/or “specific interest” in antineoplaston therapy, Livingston therapy, Evers therapy, 714x, Issels’ whole body therapy, Hoxsey therapy, hydrazine sulfate, Revici therapy, tumour vaccines, shark cartilage, Di Bella therapy, Kelly therapy, Laetrile, Gerson therapy, Essiac, Immuno-Augmentative Therapy (IAT), American Biologics therapy, Iscador, Macrobiotics, oxygen therapy, hyperthermia, chelation therapy, DMSO therapy, live-cell therapy, Aboriginal therapies, vitamin therapy, laser resonance, bio-electric therapies, herbal therapy, and Nieper therapy. No method in the “specific interest” list has been proven effective.
Previous versions of CCRG’s Web site have claimed that CCRG “has developed the world’s largest cancer knowledge base, Cancer Canada Net™, fielding over 50,000 inquiries a year, and with an active case load of more than 5,000 international patients.” [1] I seriously doubt that CCRG could have assembled “the largest cancer knowledge base.” Even if O’Neill had managed to compile citations from the major cancer-related databases, the information would not be very useful without interpretation by a qualified cancer specialist. Moreover, these databases can be accessed free-of-charge on the Internet.
In 2006, Canadian Television’s W-FIVE team aired a four-part investigative report that amply demonstrated O’Neill’s dishonesty. One woman, Teresa Bagyan, whose husband Frank had visited CCRG in 2003 after being diagnosed with a brain tumor with a dismal prognosis, told the team that “O’Neill “looked Frank straight in the eye and said ‘I’ve never lost a patient yet.'” She also told the team that O’Neill had also looked at a “before and after” set of MRIs and claimed that the immune system was attacking the tumor and “breaking it up”. After Frank’s death, a radiologist looked at the same MRIs and said that the tumor was still there at the time, with no sign of shrinkage. (That sounds a lot like Dr. Stanislaw Burzynski’s grift.) As part of the investigation, two producers, one of whom was pretending to have stage III lung cancer, visited CCRG with a hidden camera and recorded O’Neill claiming that CCRG’s cure rates were over 80% for lung cancer and 100% for lymphoma, leukemia, and prostate cancer. O’Neill later refused more than half a dozen requests for documentation supporting those claims.
You get the idea.
Ultimately, CCRG was renamed Immune System Management (ISM), a rebranding that was occurring around the time Mr. O’Neill threatened me. Mr. O’Neill ultimately died in 2013, but his son Liam appears to be running ISM now.
In retrospect, I do love the part about not distributing or reproducing O’Neill’s email without permission. Back then, I was so green that I didn’t know that that line is almost always pure unenforceable BS. These days, when I receive emails with legal threats like the one above, I’m fairly likely to post them to Twitter, Facebook, or one of my blogs. But back then, I freaked out, especially since O’Neill CC’ed my department chair, my cancer center director, and my division chief at The Cancer Institute of New Jersey and UMDNJ-Robert Wood Johnson Medical School (since renamed The Rutgers Cancer Institute of New Jersey and the Rutgers Robert Wood Johnson Medical School). I had started the blog under a pseudonym because, if anything, writing a medical blog back then, specifically a blog dedicated to combatting pseudoscience in medicine, was looked on a lot less favorably then than it is now, and, as you’ll see, there is still a sizable segment of medical academia that doesn’t look very favorably on it even now. After receiving the email, I sent late night panicked emails to my bosses explaining what was going on.
Fortunately (I guess), the responses ranged from puzzled to supportive to indifferent. Dr. Stephen Lowry, who was my department chair at the time, was supportive. (Sadly, Dr. Lowry died suddenly of an aortic dissection in 2011, three years after I had moved on from the department. He was an amazing surgeon-scientist and man.) Indeed, at a department meeting later that month he told me that people like O’Neill were bullies, which is true, although when I mentioned that I wanted to hit back by writing about him more he advised me against it. So, even though he was supportive, he wasn’t that supportive. In retrospect, he was probably correct, as Bowditch let me know that O’Neill was all bluster. Hilariously, he even cc’ed me on an email to Mr. Taller asking if he was actually representing O’Neill. Apparently, O’Neill was just adding Mr. Taller’s name to make it appear that he had legal representation.
I laugh at this early attempt at intimidation now, but at the time I was not laughing, and I can see how a newbie trying to get started now at what I was getting started at then might, facing such threats plus online harassment, decide that it’s just not worth it. Since then, as I mentioned above, I’ve received much worse. It seems that, every several months to a couple of years, something happens, most often antivaxxers or cranks trying to harass me at work by complaining to my bosses about imagined misdeeds. Most recently, antivaxxers threatened to come down in their full cosplay gear with their misinformation-packed signs to “protest” against me at my cancer center (see the comments of this Facebook post). No one’s shown up yet, but administration was concerned enough to have me make a report to hospital security. I suspect they won’t show up because even antivaxxers know that disrupting activities at a cancer center and potentially disturbing or interfering with the treatment of cancer patients is not a good look, but who knows? I also note that my cancer center director was totally supportive. He’s come a long way, having gone from being puzzled and (to my perception at the time, at least) seemingly annoyed by such complaints when he first took over ten years ago to being very supportive. He even jokes with me at times, saying that he hasn’t gotten any complaints about me recently, to which I usually joke back with a response something like, “I must be falling down on the job then. I promise do better.”
The harassment climate gets worse
McKie’s article goes on to recount a list of scientists targeted for harassment, including:
    • Climate scientists whose emails were hacked at the University of East Anglia. The hackers used the emails to paint a false picture that climate science had been manipulated and that scientists had tried to silence critics.
    • Naomi Oreskes, professor of history of science and affiliated professor of Earth and planetary sciences at Harvard University and co-author of Merchants of Doubt: How a Handful of Scientists Obscured the Truth on Issues from Tobacco Smoke to Global Warming. She recently recounted how since 2004 she’s been attacked for defending the scientific consensus regarding climate change.
    • Hero Saharjo, professor of forest protection at IPB University in Indonesia. He is an expert on the forest fires that regularly sweep through Indonesia, started by palm oil companies that want to clear land cheaply and quickly. He noted that during four months in 2015, such fires released more carbon dioxide than the entire European Union, and through his studies, has “traced those who started those fires and has testified in hundreds of cases against the companies responsible”. He and his family have faced intimidation and he’s had little support among his colleagues, many of whom are funded by the palm oil companies.
    • Simon Wessely, professor of psychological medicine at the Institute of Psychiatry, Psychology and Neuroscience at King’s College London. An expert in chronic fatigue syndrome who’s shown that two approaches, graded exercise therapy and cognitive behaviour therapy, can help CFS patients manage their symptoms, he’s been falsely attacked by patient groups for not treating CFS as a serious illness. In one instance, a conference slide of his was altered to reverse the meaning.
    • Sameer Jauhar, a senior research fellow in the department of psychological medicine at King’s College London. According to McKie: “After speaking on BBC Radio 4 about the dangers of overstating the withdrawal symptoms from antidepressant drugs, ‘people on Twitter accused me of not caring about patients, saying I was frying people’s brains’, he says.”
    • Edzard Ernst, emeritus professor of complementary medicine at the University of Exeter. We’ve long admired Prof. Ernst for his work against alternative medicine pseudoscience, and for standing up to the many attacks, including attacks by the powerful, such as Prince Charles.
    • Riko Muranaka, a Japanese scientist at Kyoto University. She was “She was threatened, sued and accused of being in the pay of the pharmaceutical industry after she wrote an article claiming that a mouse study revealing a link between the [HPV] vaccine and brain damage was fabricated.”
    • David Nutt, Edmond J. Safra professor of neuropsychopharmacology at Imperial College London. He was fired as chair of the UK Home Office’s Advisory Council on the Misuse of Drugs for saying that “alcohol and tobacco were more harmful than LSD, ecstasy or cannabis.”
    • Linda Bauld, the Bruce and John Usher professor of public health at the University of Edinburgh. She’s been hounded by Big Tobacco for her work exposing the dangers of tobacco and more recently accused of being in the pocket of Big Tobacco for her work suggesting that e-cigarettes are less harmful than cigarettes. She’s suffered online trolling and anonymous threatening phone calls, while activists are “digging into Bauld’s life and work, she says: ‘They find meetings I’ve attended to see if someone from the tobacco industry was there so they can say I am biased’.”
Reading McKie’s article, I see some things getting worse. For instance, the online harassment is definitely getting worse, from my perspective. I suspect the reason is that it’s so much easier to harass someone on social media now, and denialists are so much better organized than they were back in 2005, when I suffered my first brush with such tactics. Think about it. Back then, Twitter (founded in 2006) and Instagram (founded in 2010) didn’t yet exist; Facebook was still limited to college students at Ivy League universities and nearly a year and a half away from being made available to everybody. The only venues for online harassment and the organization of campaigns of online harassment were, essentially, email, blogs, and Usenet, and Usenet was fast dying, thanks to the rise of web-based discussion groups. Twitter, in particular, can be a real cesspit, particularly if you’re a woman or person of color, so that the online attacks you face can have misogyny, threats of sexual violence, and racism added to posts like this:
You might recognize that name. He’s commented here and on my other blog in the past.
Similarly, the rise of social media has made it much easier for denialists to organize harassment campaigns. There are secret Facebook groups (and ones that aren’t secret) where members plot things like protesting at my cancer center. Back in 2010, when my university was the target of campaign by antivaccine activists, I knew that the campaign had largely been organized by email. Today, it would have been organized on Facebook, in one of the many antivaccine groups that still exist even after Facebook’s crackdown on antivaccine misinformation, where thousands more people could be reached.
Moreover, if you work for a private company, do you think your bosses will be supportive if the online mob targets you for harassment at work? I’m fortunate in that I work in academia. The culture here is very much more resistant to online mobs like this than private industry. If you work for a private company and are targeted, there’s a high likelihood that your bosses will tell you to shut up or be fired, not because they aren’t sympathetic but because they don’t want to be bothered, they don’t want the company’s reputation to suffer, and they don’t want controversy involving the company. There’s even a little of that in academia, but freedom of academic expression is so strongly ingrained in the culture of universities that it usually wins out, bringing at least grudging support from administration and in my case (usually) enthusiastic support.
There is, of course, one drawback if you work for a public university. It’s one I’ve discovered before, namely that you might be subject to Public Record Act/Freedom of Information Act (FOIA) requests. It’s a tactic antivaxxers are using increasingly. I’ve been targeted by Gary Null, and Dorit Reiss has been targeted by Del Bigtree. In fact, I’ll launch a preemptive strike here against Null and tell you what was sent to him. Amusingly, Null will get a couple of crank emails sent to Steve Novella and me at our university addresses. (That’s what he gets for asking for any emails mentioning Steve Novella.) He’ll also get an email chain among several provaccine activists discussing the proposed Connecticut law to eliminate religious exemptions to school vaccine mandates. My prediction is that one email in particular will be publicized by Gary Null, in which I wrote:
This is why I find religious exemptions for vaccine mandates so problematic. They unjustifiably privilege one form of personal belief over another just because…religion. There is no meaningful distinction between religion and other personal beliefs, any of which can easily be rendered bulletproof for purposes of these laws by claiming a religious basis.
A couple of those on the email exchange agreed. Of course, what I meant was that exemptions based on religion only are inherently unfair because they discriminate against nonbelievers and unfairly privilege religious belief over other deeply held ideological/spiritual beliefs, which is why I’ve always believed that if you’re going to allow religious exemptions you have to allow personal belief exemptions and that if you’re going to get rid of personal belief exemptions to be fair you have to get rid of religious exemptions as well. It’s both or neither. Of course, I also believe that it’s also much better to get rid of both religious and personal belief exemptions when politically feasible and allow only medical exemptions. Gary Null is very much into labeling anyone who questions his quackery as “atheists” (as if being an atheist were a bad thing!); so I fully expect he’ll try to make a stink about my statement, proclaiming it as evidence that I’m an evil atheist targeting “deeply held religious beliefs”. If you see him do that, feel free to refer him to this post.
The climate improves
It’s not all bad news. In many ways, as McKie recounts, physicians, scientists, and academics are more supportive now when colleagues come under attack. There’s even a prize, the John Maddox Prize, awarded by Nature and Sense About Science every year to individuals who promote science and evidence in the face of hostility. Bambang Hero Saharjo is the recipient of this year’s award. Several others mentioned above have also won the award, including Edzard Ernst, Riko Muranaka, and Olivier Bernard, who won this year’s early career Maddox Prize. I’ll let you in on a secret. I’ve been nominated on at least a couple of occasions. (Thanks to those who nominated me.) Looking at the winners, I realize that I probably haven’t faced the level of high-power hostility that those who actually won have. Personally, I like to think that campaigns against me over the years have led several of my bosses to appreciate the importance of standing up for science as academics, leading to more enthusiastic support. (It also doesn’t hurt that the online cranks who launch these campaigns are often so…cranky. They make it obvious what they’re about.)
Other areas in which the climate has improved include efforts by Google and social media giants like Facebook to rein in the spread of misinformation about medicine, at least. Doctors and those who support science-based medicine have finally woken up and started to organize to help those who are targeted by quacks and antivaccine cranks. The most prominent example is Shots Heard, which exists mainly to counter the swarms of antivaccine activists who will infest the Facebook pages of doctors who promote vaccination and publish fake negative reviews about them on Yelp! and Vitals.com. Such efforts are still fairly embryonic and small, however. Supporters of SBM remain outnumbered, outgunned, and outfunded, but things are getting better.
Except for one thing.
The final challenge to science communication
There is one last major impediment to the growth of science communication to the point where the battle against pseudoscience doesn’t seem so hopeless. There remains a prevalent attitude out there among some academics, at least one of them quite prominent, that countering quacks and antivaxxers is so easy that it’s not worth our time as physicians. One major example with whom I tussled on Twitter is someone whom I like to call a Very Eminent Rising Star in Oncology (VERSO)—mainly, because he is just that—Vinay Prasad, MD, MPH.
Back in October, he started Tweeting about how criticizing Goop’s selling of jade eggs and other pseudoscience was like “dunking on a 7′ hoop” and how skeptics should put the hoop back up to 10′ by—of course—joining him in taking on clinical trial shenanigans by big pharma. He even went so far as to suggest that I give up my usual topics for six months and spend those six months doing what he does because what he does is so much harder and more worthwhile. He appears to have since deleted many of the most obnoxious of the Tweets in which he made fun of skeptics who spend time deconstructing quackery and pseudoscience, implying that such activity is so easy as to be unworthy of our efforts and that what he does, analyze clinical trials, is so much more difficult. Of course, there’s considerable overlap in skillsets, given that we frequently have to go deep into the weeds of clinical trials of, say, acupuncture or homeopathy to demonstrate why they do not show what advocates think. When this was pointed out to him, even more obnoxiously, he took it as confirmation that we think that what he does is of more value because we were arguing that we do do the same sorts of analyses of clinical trials that he does. Be that as it may, I tried to search Twitter for the most offensive posts, and it seems that he’s deleted many, perhaps all, of them.
Here’s a remaining example someone took a screenshot of:

In context, he was clearly referring to me (the “fellowship trained” bit). The “soft targets” bit is a claim that he repeated several times. It’s one I’ve dealt with before elsewhere.
Basically, Dr. Prasad raised criticisms of our efforts as though he were the first to think of them and was very dismissive, thinking his efforts far more difficult and worthy than ours. In any event, I don’t want to go into detail about the whole kerfuffle. I’m merely using it as an example of how dismissive some physicians, particularly academic physicians, are of science communication efforts targeted at quackery. I liked Mark Hoofnagle’s comments, though:
I’m sorry, but I’m about to start cursing.
Do you think maybe the people who have been doing this for a decade maybe know about the backfire effect?
Did you Dunning Krugers stop for a second to consider asking one of us about it?
Oh jeez, ya think?
— Mark Hoofnagle (@MarkHoofnagle) November 2, 2019
Didja think to ask any one of us, like me, who was linking science of scicomm over and over again whether or not there was evidence or a literature that we used to back our technique? Or did you, like Vinay, just stop at the most basic fucking problem in debunking?
— Mark Hoofnagle (@MarkHoofnagle) November 2, 2019
I mean, it’s not like I was linking papers on advanced tactics like inoculation, or discussion how you specifically respond to disinformation.
Have you read the handbook? Yeah, there’s a fucking handbook. https://t.co/GdzHGUlp5O
— Mark Hoofnagle (@MarkHoofnagle) November 2, 2019
Christ on a cracker. You people, you’re so dismissive, you didn’t listen, but managed to convince yourselves you’re so smart for thinking of a problem we’ve been addressing for a decade.
— Mark Hoofnagle (@MarkHoofnagle) November 2, 2019
If it’s not clear I’m not directing this at @JHowardBrainMD but at the unbelievably shallow assholes that think theyre smart for finding the most obvious question in scicomm and posing it like it’s a novel insight.Maybe next you’ll tell us the information deficit model is wrong!
— Mark Hoofnagle (@MarkHoofnagle) November 2, 2019
That was basically my attitude, too. Maybe I should go back and analyze Dr. Prasad’s podcast on the issue, but I fear it might raise my blood pressure too much. The point is simply how dismissive some doctors are of science communication efforts targeting pseudoscience, even as they “dunk” on a clinical trial based on…a treatment from alternative medicine:

The point is that, yes, things have gotten better in that many more physicians and scientists are communicating science and pushing back against pseudoscience than there were when I really got started 15 years ago. Also, it should be a big tent, with room for doctors like Dr. Prasad and their critiques against pharma and doctors like us at SBM. We should be natural allies, as is the case with Ben Goldacre, who takes on both pharma and alternative medicine. To me, that’s the hardest nut to crack. Until we manage it, until we can persuade doctors like Dr. Prasad that we’re natural allies and that combatting pseudoscience is not only difficult but a worthwhile activity for an academic, we’re going to be held back in what we can do. Doctors like him don’t have to do the dirty work of pushing back against quackery themselves if they don’t want to. Division of labor is a good thing. All I ask is that they stop making our job harder.
As for everyone else, doctors, scientists, and interested lay people, we welcome you. I just wanted you to know what you could be in for if you join is, so that you go in eyes open. Only you can decide if helping to educate the public and counter disinformation is worth it to you.

Why Science-Based Medicine Matters
James Thomas on June 25, 2016
Link


The regular contributors at Science-Based Medicine (SBM) work diligently every week to explore the world of science-based medicine and the gauzy, nebulous netherworld of fantasy-based medicine. They shine light on the leading edge of medical science, dissect the nuances of mainstream care, expose the misconceptions and sometimes the frank deceptions of so-called alternative medicine. Launching SBM on January 1, 2008, sbmadmin (Steven Novella?) described the mission of the blog as: “scientifically examin[ing] medical and health topics of interest to the public [including] reviewing newly published studies, examining dubious products and claims, providing much needed scientific balance to the often credulous health reporting, and exploring issues related to the regulation of scientific quality in medicine.” He went on to propound an elegant yet simple core philosophy that “safe and effective health care is critical to everyone’s quality of life; so much so that it is generally considered a basic human right.”
This last deserves, I think, especially careful consideration. Enshrined in the aspirational manifesto of the United States, the Declaration of Independence, is the claim for all people to inherent and inalienable rights to “Life, Liberty and the pursuit of Happiness.” And while Jefferson used the phrase in the context of personal political freedoms, health is central to the exercise of those rights. It is the role of medicine to secure health, to provide a structure and a system by which all can live life as long and as free from disease and disability as our individual circumstances allow.
What came before
Our efforts to formalize the management of human health date back long before Hippocrates and, several centuries after him, Claudius Galen who codified the sum of what then passed for medical knowledge in the West. Ayurveda claims a history of 5,000 years, Traditional Chinese Medicine at least 2,500 years, and other cultures from Babylon to Egypt had medical traditions of varying degrees of formalization. But these were medicines of speculation, of humours and miasmas, of herbs and poultices, of pakrutis and meridians and qi. What is remarkable is how little these traditional medicines changed over the centuries. It isn’t that medicine was static for several thousand years, but change was incremental, stifled by unchanging conceptual paradigms, and boxed in by cultural and religious mores. For instance it wasn’t until the 13th century that an Arab physician named Ibn al-Nafis discovered that blood moved from one side of the heart to the other, first traveling through the lungs to be “mixed with air.” This overturned the Galenic belief of a porous cardiac septum, a belief held since the 2nd century.
Centuries of fits and starts of incremental progress did little to curb the major causes of morbidity and mortality. Roman babies died of pertussis when Julius Caesar declared himself emperor. Chinese babies died of pertussis in the 7th century. North American babies still died of pertussis in the early 20th century. For that matter, American babies are still dying today. In fact, this might be a good point at which to mention that it is the very young who historically have paid the steepest price for ingrained and refractory medical ignorance.
We often cite changes in life expectancy as a measure of medical progress. But as we shall see, that is a remarkably crude measure. Selected males born before 100 BCE lived a median span of 72 years , not remarkably different from lifespans in industrialized nations today. But it is important to note that the data included only males who lived long enough to make a contribution of sufficient significance that their vital data was recorded in the Oxford Classical Dictionary. Looking at changes in life expectancy for women aged 15 and older reveals a very different picture. In the 16th century these women had a life expectancy of about 48 years. Much of that premature death owed to maternal perinatal mortality. It is an interesting historical side note that it was puerperal fever (“childbed fever”), a frequent cause of maternal death, that led Ignaz Semmelweis to his work on antiseptic hand washing – part of the cascade of discoveries that gave birth to the germ theory of disease.
Pity the child
Before one can conceive or for that matter father a child, before one can make a contribution worthy of history’s notice, one must generally survive childhood. Until very recently, that was in no manner a certainty. It is very difficult to find meaningful numbers for infant mortality rates in 700 BCE China or 100 CE Rome or 800 CE Arabia. There are a variety of numbers bandied about but in the 17th century the consensus seems to be that about 30% of English children died before the age of 15, mostly from infectious diseases. Factoring in the ghastly premature deaths of children and mothers, average life expectancy in the time of the Roman Empire was perhaps 35 years. As late as 1900 in the US, life expectancy at birth for males was 46.3 years and 48.3 years for females. But if you lived to see your 20th birthday your life expectancy (white males) was 62.2 and (white females) 63.8. So when you read statistics of life expectancy in the days before modern medicine, remember that those short spans didn’t so much reflect a truncated dotage as a failure to survive childhood – or even the miracle of childbirth.
Rubella, measles, diphtheria, smallpox, puerperal fever, the odd plague, scarlet fever, pneumonia, tuberculosis, and a host of other diseases thinned the herd of infants and children. And whether in China or Africa or Asia or Europe, traditional medicine treated these diseases with herbs and needles and spells and prayers. And thank goodness! Because with all of those “interventions” only about 30% of children died. Of course only about 30% would have died without the interventions. And given some of those treatments – bleeding comes first to mind – perhaps doing nothing might have yielded slightly better mortality rates.
But in the eyes of the times, recoveries were miracles of what then passed for medicine. Chen Liu was ill. The traditional healer came. After some days, Chen Liu was better. Give thanks to the traditional healer! This is a classic example of confusing correlation with causation. The human body maintains a wide variety of parameters within very narrow limits: core temperature, blood pH, partial pressures of oxygen and CO2, and very many others. The balance achieved is referred to as homeostasis. When one or more of these parameters drift out of normal range, various biological processes work to restore homeostasis. When disease or injury disrupts homeostasis, one of two things happens: either homeostasis is eventually restored and you get better (sometimes referred to as regression to the mean) or you die. The mechanisms that directly and indirectly enforce homeostasis are very powerful indeed, so regression to the mean is, generally speaking, the usual outcome. But with disease or injury there is often uncertainty. So when a shaman or healer intervenes and the patient recovers, it is easy to credit the intervention. And when the patient dies it is because the shaman wasn’t summoned soon enough.
Belief is powerful, yet powerless
And so it goes. Or so it went. Disease was caused by unbalanced humours, miasmas, disrupted qi, evil spirits, or whatever you did to piss off the gods. Healers responded with the tools they had. They poked needles to unblock qi, they bled patients to balance humours, they prayed to the offended gods, they applied poultices and prescribed herbs. The common thread is that all of these approaches were based on beliefs rather than knowledge, and those beliefs were absolutely, flatly wrong*.
Belief is a powerful thing – and all too often an easy thing. The beliefs of our forebears are often unquestioned and comfortable. The first effort to conceptually challenge blind faith in received wisdom was perhaps taken by Thales of Miletus, a 7th century BCE Ionian philosopher who worked to understand the world in natural terms without relying on mythology or supernatural explanations. Reason and rationality again flourished for a time during the Islamic Golden Age with significant contributions to mathematics and astronomy and, as mentioned earlier, even medicine. There were brilliant insights and discoveries throughout these long centuries, but they were, by and large, naked flashes against a dark fog of long cherished beliefs, unquestioned religious dogmas, and primeval superstitions. What there wasn’t, was science, at least not in the sense of a broadly organized effort to understand reality.
Beliefs die a hard death. By way of illustration, On the Origin of Species was published in 1859. The Scopes “Monkey” trial was held in 1925. The US Supreme Court ruled against teaching Creationism in Edwards v. Aguillard in 1987. Over the years an imposing body of evidence supporting evolutionary theory has been assembled. Yet still in 2016, long past Origin’s sesquicentennial, groups such as The Discovery Institute still peddle a version of creationism they style as “Intelligent Design.” Despite the overwhelming mass of evidence supporting evolution, despite virtual unanimity of scientific consensus, a Gallup poll in 2014 found that 42% of Americans believe that “God created humans in present form.” Another 31% allowed that humans evolved, but with God’s guiding hand. Only 19% understood that humans evolved through natural processes and without necessity of supernatural intervention.
The fog of dogma and superstition began to lift as the Enlightenment rolled across Europe in the 18th century. Blind faith – in religion, in monarchies, in hidebound beliefs long sheltered from doubt – was shaken by a new willingness, a new impulse to question and reexamine. By the 19th century the millennial shackles of medical thought began to crumble as Schwann and Schleiden described their theory of the cellular nature of living tissue, Jenner introduced smallpox vaccination, Lister built on the work of Pasteur by introducing antisepsis in the form of carbolic acid to sterilize surgical instruments and surgeons’ hands. It was clear that the medicine of Galen, a medical tradition that had endured for well over one thousand years, was crumbling.
The inevitable backlash to reason
But then, as now, logic and reason are not everyone’s cup of tea. The profound challenges unleashed by the Enlightenment also triggered the Romantic backlash of the 19th century and with it renewed interest in vitalism and, especially in North America, spiritualism and religiosity. Mormons, Seventh Day Adventists, Jehovah’s Witnesses, Christian Science, and other religions sprang to life. And efforts to fill the void of the rapidly collapsing Galenic tradition were not limited to scientists.
Samuel Hahnemann propounded homeopathy around 1800. Hahnemann, a medical doctor from what is now Germany, was troubled by the increasingly obvious flaws of Galenic medicine. He recognized that the treatments he learned in medical school often seemed to do more harm than good and he quit the practice of medicine within a few years, earning his living as a translator and writer. While translating Cullen’s A Treatise of the Materia Medica Hahnemann encountered the claim that the bark of the South American cinchona (Cinchona officianalis) was an effective treatment for malaria because it was a bitter astringent. Hahnemann doubted that astringency alone accounted for cinchona’s effectiveness, questioning why any other astringent wouldn’t have the same effect. Hahnemann, though not suffering from malaria, decided to experiment with cinchona on himself and found that ingesting some the material produced symptoms which he imagined to be malaria-like. From this he extrapolated the dogma of “that which can produce a set of symptoms in a healthy individual, can treat a sick individual who is manifesting a similar set of symptoms.” This is the “like cures like” principle on which homeopathy is based. This massive leap of intuition was, and remains, spectacularly wrong.
You will note the ‘sciencish’ impulses that lie within Hahnemann’s journey. He recognized serious problems with Galenic medicine as it was practiced in his time, he questioned an understanding of a then mainstream treatment for a serious disease, and he attempted, albeit ineptly, to do an experiment to improve his understanding of that treatment. Unfortunately, the whole enterprise was built on the fantasy that it was the astringency of the cinchona bark and its “tonic effects on the stomach” that gave it therapeutic efficacy. We now know that cinchona bark contains quinine among a variety of other compounds not all of which are benign. In the event, what Hahnemann constructed was nothing more than a classic argument from false premises – and he attempted to build an entire medical paradigm around it.
Flexner and the founding of American scientific medicine
Hahnemann was not alone among 19th century medics to behold the many shortcomings of medicine and to propose a new approach. Near the end of that century DD Palmer proposed chiropractic (and for a time considered establishing it as a religion) and Benjamin Lust founded the panvitalist cesspool of naturopathy. All were creative and not one was scientific in any meaningful sense of the word. Each replaced one set of fantasies with another. Physicians of that era gleefully picked and chose from that cornucopia of delusion to practice as they best saw fit. The practice of medicine grew more diverse but precious little grew better. By 1904, the American Medical Association had had enough and established the Council on Medical Education to bring some order to the chaos. In furtherance of better medical care, the Carnegie Foundation, supported by the AMA, commissioned Abraham Flexner to study medical education in the United States and Canada. The resulting Flexner Report [PDF] eviscerated medical education as it then existed in North America and resulted in the closing of a large portion of the medical schools then operating. Flexner was himself not a doctor, but would probably today be recognized as an expert in pedagogy. Why a pedagogue you ask? As Thomas Duffy notes:
Flexner was an unorthodox and surprising candidate for the task he was asked to undertake. Flexner himself was quizzical about the summoning, suspecting that he was being confused with his brother, Simon. At the time of the job offering, the former high school teacher had never been in a medical school. This shortcoming might have seemed an insurmountable impediment for successful performance of his assigned task, but the choice of a non-physician was purposeful on the part of Pritchett and his associates. They perceived the problem of medical education as a problem of education and believed a professional educator was better qualified to address this dimension of the problem. They also had preconceived ideas concerning what changes needed to be made in medical schools to allow these ideas to be introduced. The ideas Flexner popularized were those that had already been developed within medical schools before the turn of the century. Pritchett (Henry Smith Pritchett was president of the Carnegie Foundation from 1906-1930) and colleagues also were concerned that antagonisms would be generated by the report, which might be less vengeful if a non-physician were the object of the resentments. An unflattering but not necessarily inaccurate description for Flexner’s assignment was that he was to be the hatchet man in sweeping clean the medical system of substandard medical schools that were flooding the nation with poorly trained physicians.**
Flexner and his Report, of course, did not change medical education single-handedly; his contribution should not be oversold. In important ways he was simply a reporter of facts – and often a lightning rod. Giants including William Osler and William Welch (two of the four founders of Johns Hopkins Hospital) shaped the contours of what would become modern medical education. But neither should Flexner’s role be undervalued. He laid bare a medical education system that was outdated, inconsistent, and mostly ineffective. And he promoted the “German System of Medical Education” and its program in which “physician scientists were trained in laboratory investigation as a prelude and foundation for clinical training.” This was the seed from which modern medicine, science-based medicine, took root.
Before Flexner, medicine had changed only incrementally since the time of Galen – the better part of 2,000 years. In the hundred years since Flexner, medicine has been radically transformed and with it the human condition, from the Hobbesian “nasty, brutish, and short” to a span rapidly closing on nine decades of often quite good health. While homeopaths shook their little vials of magic water, science-based medicine eradicated smallpox, a disease that as recently as 1948 infected 50 million people worldwide. Chiropractors imagined subluxations were the one true cause of all “dis-ease.” They adjusted and twisted and prodded while science-based medicine learned to type and transfuse blood and conquered, for instance, most childhood leukemias. Naturopaths flushed pounds of imaginary ‘toxins’ from colons, while science-based medicine identified the virus that leads to AIDS and developed Highly Active Anti-Retroviral Therapy to manage the disease.
The turning point: A method, not a fact
The inflection point on which this revolution turned was not a powerful new type of microscope or a brilliant new surgical technique or a dramatic insight into how some organ system worked. It was a change in the manner of thinking. No longer would intuition and belief and the received wisdom of our forebears be enough. Henceforth these would be hung, along with new insights and developments, on the cold rack of the scientific method and rigorously tested. This was the moment when the paths of medicine and the array of fantasies and delusions that now self-identify as “integrative medicine” parted. Medicine’s commitment to science led to the cascade of advances in understanding and application that today render obesity a far bigger concern to most people than, say, diphtheria.
Science works. It delivers the goods. In 1900 the top causes of death were pneumonia, tuberculosis, and diarrhea and enteritis. Today, accidental injury claims more lives than any of them. In 1969 the age-standardized death rate from heart disease was 520.4 per 100,000. By 2013 the mortality rate was less than half that: 169.1 per 100,000. This was not by happy accident. This was not the result of deeper understanding of chiropractic subluxations or new insights into the vital force. It was the result of the hard work of thousands of research scientists and physicians painstakingly peeling back layers of ignorance and illuminating the innermost workings of biology and disease.
The cost of medicine without science
This is not to say that quacks do not try to wrap themselves in the trappings of science. Journals of rank silliness litter the professional literature. Schools of naturopathy teach courses in microbiology and immunology. Homeopaths jabber about the quantum nature of water memory. This is not science, it is cargo cultism. The philosophical underpinning of medicine is science. The philosophical underpinnings of ‘alternatives’ to medicine are alternatives to science: vitalism, spiritualism, superstition. And let no one forget that the sum of the contributions of naturopaths, of chiropractors, of homeopaths and the rest of the alternative practitioners to the revolution in health care has been and remains … zero. Not one insight, not one discovery that has meaningfully changed the face of health care.
Imagine for a moment that Flexner and Welch and Osler had occupied themselves with astronomy or law rather than to medicine. Imagine life today with medical care delivered by homeopaths and naturopaths and chiropractors and reiki “masters”. Imagine a health care paradigm that rejects vaccinations. Smallpox would likely continue to ravage the developing world and pertussis infections could still kill ten thousand children each year. Imagine a health care paradigm that holds human health to be regulated by qi or by the ‘vital force’ revered by naturopathy. Childhood leukemias would still be death sentences. Liver disease would be treated with spinal adjustments and diabetes treated with colonics and macrobiotic rice. In short, imagine that alternative medicine is the only medicine we have. That was largely the reality until medicine chose the arduous path of science over the easy embrace of custom and convention.
That is why science-based medicine matters. Medicine without science is the medicine of Galen. It is the medicine of bleedings and herbs and colonics and prayer. Science-based medicine is the germ theory of disease, it is antibiotics and antivirals, it is kidney dialysis and cardiac pacemakers. Science is the flashlight that reveals the wires and the gears and the pushrods of human biology. Alternative medicine is the opposite of a flashlight. It is a flashdark that substitutes fear for knowledge, superstition for understanding, ignorance for insight.
Immanuel Kant saw enlightenment as “man’s emergence from his self-imposed childishness.”*** We stand today in witness of a revolution in our understanding of biology. We have exposed the workings of the very recipe of life and are rapidly learning the often-interconnected functions of various genes, the implications of this and that allele, the secret lives of some proteins. We are building the tools to reshape malfunctioning genes, viruses to carry new instructions to cellular machinery, systems to build entire new organs.
When I was a child, I spoke as a child, I understood as a child, I thought as a child; but when I became a man, I put away childish things. For now we see in a mirror, dimly, but soon face to face. Now I know in part, but soon I shall know just as I also am known. Corinthians 13:11. Kinda
It is past time for us to put aside our childish things. A future of better health and happier lives lies in our deepening understanding of our biology. This understanding will be found in electrophoresis gels and chromatograms and the tireless work of thousands of scientists and physicians, not in color flecks in irises or the memories of water molecules.
Notes
*Of all this nonsense, sometimes the herbs had actual biological effects and sometimes those effects were even useful in treating the disease the patient actually had. But diagnosis was imprecise and the dosage of herbs less precise still. The marrying of effective dosage of herb and disease for which it was appropriate was largely a matter of serendipity.
** I would like to acknowledge that regular SBM commenter and moderator William Lawrence Utridge drew my attention to some of the controversy surrounding the Flexner Report.
*** Kant uses the word ‘Unmündigkeit’ which has been most often translated as ‘nonage’ or ‘immaturity.’ I argue that the word ‘childishness’ more fully conveys Kant’s intent.

Shaping Opinions About Science and Medicine
Responding to a commentary with all the usual anti-science tropes.
Steven Novella on July 24, 2019
Link

The alternative medicine industry has been remarkably successful in shaping public opinion about the nature and relationship of science and medicine. A half century ago much of what is now considered by the public to be a valid alternative, or something that should be “integrated” into mainstream medicine, was considered to be health fraud. Like any complex social phenomenon, some of this rebranding was deliberate and calculating, and some emerged spontaneously from the community. The result has been a set of specific misinformation about science and medicine that works together to undermine the scientific position and promote quackery as legitimate.
I was recently sent this editorial, which nicely summarized that very position, and asked to comment. It is from a local news outlet in Alexandria, MN, but it could have easily had appeared in a big city newspaper, and I have encountered some version of this position countless times. The headline nicely encapsulates this position – “It’s Our Turn: Proof vs. possibilities”. The phrase, “It’s our turn” clearly stakes out the personal freedom and empowerment position that is common among so-called CAM proponents. It is the second part, however, the “Proof vs possibilities” that is the core of the piece, and represents a false dichotomy and significant misinformation at the heart of CAM apologetics.
The false dichotomy is the notion that mainstream medicine is too demanding and unreasonable when requiring scientific proof for treatments, while being hostile and dismissive of the possibilities of treatments outside the mainstream. This is a powerful narrative, but it is all nonsense. It is, to be honest, partly a manifestation of the Dunning-Kruger effect. The author appears to be writing from the illusion of knowledge, while demonstrating they don’t actually understand how modern medicine operates. As is often the case, there is a small kernel of truth lurking deep within some of his points, but they are distorted beyond recognition to serve a narrative that has evolved to promote a brand.
So let’s tease apart his specific points, which will be familiar to anyone engaged with science-based medicine, and examine them. He writes:
But it’s not easy staying healthy, especially when you consider all the contradictory and confusing health advice out there. One expert tells us to do one thing and another tells us to do another. Sometimes it works, and sometimes it doesn’t.
This is a clever way to dismiss or at least downplay expertise. Those experts often disagree with each other, so we can’t trust them. This is part of “The Death of Expertise” that has been marshaled increasingly recently in service to all sorts of unscientific positions. It is also a common strategy of the doubt and confusion used by science deniers. I am not saying that the author is a denier, just that this is a common anti-science strategy. Again – there is a kernel of truth here. Experts don’t always agree, especially in complex areas where our knowledge is limited. That does not mean, however, that there is always a lack of consensus.
Some scientific and even medical knowledge is very well established, and experts will therefore tend to agree (and the majority of health advice is actually pretty simple). I am often the second or third opinion for the patients I consult on, and they often tell me that my opinion is exactly what the previous neurologists also told them. Sure – because it’s the well-established standard of care. Also, sometimes differing opinions amount to nuance, or equivalent valid alternatives, or different ways to balance competing interests. Medical experts can make different recommendations, but both still be essentially correct or practicing within the standard of care.
Experts will tend to disagree when the science is unclear and opinions hold sway. Different specialists may also disagree because they have different sets of expertise. In my experience, the source of “disagreement” is mostly just a misunderstanding on the part of the patient. Another specialist may have referred a patient to me to see if a patient has a condition, not because they thought he does have the condition. And of course experts don’t know everything and sometimes one or more are just wrong. This may be because the community is wrong – we just don’t have the knowledge – or it may be because that one practitioner lacks information or made a mistake.
So valid experts usually do agree, and when they seem to disagree there are a variety of complex reasons. But you can easily wash over all this complexity to create doubt and confusion about the legitimacy of experts or expertise itself.
One of the most mind-boggling aspects of modern medicine is its scepticism and dismissal of anything that is not part of the mainstream medical establishment.
At first glance that may seem logical, because we think medicine is supposed to be based on science.
There is nothing mind-boggling about skepticism toward unscientific or implausible treatments. This is because modern medicine is supposed to be based on science. But the key to his statement is the word “dismissal”. CAM proponents try to dismiss the skepticism of scientists as being dismissive, when it isn’t. If anything, I am critical of my colleagues for not being skeptical enough. They are anything but dismissive – they tend to be very open, even to a fault. This is simply part of the anti-science narrative, and it is not based on reality.
We have very specific logic and evidence-based reasons to conclude that most CAM treatments likely do not work. Characterizing a vast body of scientific research and careful science-based analysis as being “dismissive” is absurd, but that charge will never go away.
The problem is, medicine doesn’t always work that way. Not only is the diagnosis often inconclusive, but the treatment tends to be a drug that has uncertain side effects and often only treats symptoms and not the root cause. In addition, everyone is different and reacts to drugs differently, which makes medicine sometimes seem less like science and more like grasping at straws.
This is the “science doesn’t know everything” defense. Yes, medicine is complex, everyone is different, and we are basing decisions on statistical information. That reality is not incompatible with taking a rigorous scientific approach to medicine. It is also no different for any treatment, whether you consider it CAM or mainstream. It’s not as if CAM proponents are practicing in a different world where patients are not all different. But as is often the case, the point of such comments is not to create a coherent position, but to maximize doubt and confusion over the scientific position. It’s all chaos, so you might as well go with your feelings, and our treatments feel better.
Also notice the very common tropes that mainstream medicine is all about drugs, and that we mostly treat symptoms. Again – these are not based on reality, just pro-CAM propaganda. Scientific medicine is the only medicine which makes a specific effort to understand and treat actual causes of disease. No CAM treatment does this – because they are not based on an actual understanding of biology. Some are based on false or superstitious notions of health and illness. Others are explicitly symptom-based. Homeopathic provings, for example, are all based on symptoms. So accusing modern medicine of treating symptoms is flipping reality, but it is amazing how many people buy this.
What is also interesting is the inconsistency of this position. When we point out that CAM treatments do not address the cause of a disease and in fact have only placebo effects, then suddenly symptomatic treatment is fine. It’s the best – it improves quality of life and is patient-oriented. I have never seen this blatant double standard give a CAM proponent a moment of pause.
Doctors have a tough and important job and we shouldn’t expect miracles from them. But we should expect an openness to possibilities. Certainly there is science involved in medicine, but there is also practicality, intuition and experience. Just because something has not been proven with science yet doesn’t mean it isn’t true and couldn’t be useful to some people.
The author is trying desperately to seem reasonable, but every concession is immediately taken away. Again, this is a false dichotomy. You can be open to possibilities but also require rigorous science to test those possibilities. That is, in fact, exactly what scientific medicine is. We are, in fact, maximally open-minded. All claims are treated with an equal requirement for logic and evidence. It is the CAM proponents who are closed-minded. They are closed to the possibility that their preferred treatment does not work. No amount of evidence will convince them. They will dismiss anything that contradicts their position, and simply move on. I have an open challenge to anyone to point to a CAM modality that has been largely abandoned because evidence showed it does not work or is not safe. I am still waiting.
Physicians also don’t ignore practicality, intuition, and experience. If anything the evidence shows that we rely too heavily on these things, to the detriment of our patients. This is the whole point of evidence-based medicine, in fact. Patient outcomes are objectively better when we put evidence first. Experience and intuition (which is mostly just pattern recognition) are very helpful. They are great at generating hypotheses, but we can’t end there.
In fact, the complex relationship between intuition and analytical thinking in clinical decision making is something which is specifically studied and taught. This is again a great example of Dunning-Kruger – there is a deep and thoughtful literature dealing with intuitive vs analytical thinking in medicine, yet the author thinks they can criticize an entire profession from a position of apparently absolute ignorance about this topic.
The problem is that while medical science has no problem “proving” that an expensive drug will “safely” treat a particular symptom, it seems to have trouble telling us how much of a certain vitamin we need, or what types of food we should eat, or how to prevent disease. Worse yet, when it comes to alternative medicine, with treatments that seem to help some people, traditional medicine can be downright hostile.
The author flirts with the “Big Pharma” conspiracy throughout, implying here that we marshal our science only to prove that expensive drugs work. I will acknowledge the kernel of truth here that the pharmaceutical industry itself is largely geared toward this, but not all of medicine. There is plenty of public, private, and academic research independent of pharmaceutical companies. Doctors also use many treatment modalities other than drugs.
Why the author thinks that scientific medicine has difficulty telling how much of a vitamin people need is truly astounding. Science discovered that vitamins exist, and determine their nature, what they do in the body, what happens when we don’t get enough, and has explicitly determined within very narrow tolerances exactly how much people need. I am struggling to determine what he is even referring to, unless it is simply just more doubt and confusion generated by the supplement industry to promote unneeded megadoses of their products.
Regarding the last point, I wish mainstream medicine were more hostile toward quackery. Sadly, that is not the case. What SBM recommends is also not best characterized as hostility, but rather we simply advocate a clear and consistent standard of science and evidence to determine the risk vs benefit of any intervention. That’s it – one honest and rational standard that properly balances all interests. But it’s easy to distort this position into being “hostile” and “dismissive” when your pet theory or treatment does not survive honest scientific inquiry.
One more:
There seems to be a double standard at work here. On one hand, new drugs are approved and condoned if they appear to be safe and effective. On the other hand, herbal medicines (which are not backed by huge drug companies) are expected to prove with 100 percent certainty that they always work and are invariably safe.
This is where he goes off the rails (not that he was really ever on them). There is a double standard, but it does not work the way he claims. Drugs have the highest standards of evidence for any medical treatment. That is undeniable. The FDA requires multiple studies showing safety and efficacy, and regulates the rigor and execution of those studies closely. The system is not perfect, but it is a relatively high bar that costs tens to hundreds of millions of dollars of research to cross.
Herbs, on the other hand, have largely been deregulated (despite being essentially just dirty drugs, when they aren’t actively adulterated or contaminated). You can put an herb on the market with zero research, as long as you don’t claim to treat a disease. I wonder if the author would like to have pharmaceutical drugs and herbal drugs trade regulatory places. Or perhaps, we could just require the same bar of evidence for herbs as we do for other drugs. Deal?
I also have to wonder how they came to the conclusion that anyone is asking for “100 percent certainty” of safety and effectiveness for herbs. Where does that even come from? They seem to have made up that point out of whole cloth – but it does serve the narrative. That is a classic strawman, and I would challenge him or anyone else to point to a reference to justify that accusation.
How about we acknowledge first that herbs are drugs. Thousands of years of use does not, in fact, guarantee that they do anything (see all of prescientific medicine for a reference) or even that they are safe. Also, many modern herbal products are fairly recent, and the claim to antiquity is just marketing, or at least extremely misleading (they were often used for many vague indications, and often different from what is currently being claimed). We can dispense with the health halo, the “natural” marketing, and just treat them like drugs. Require basic testing for safety, and only allow claims based on adequate evidence.
As you can see, it takes far more time and space to correct misinformation than to create it (this is known as the Gish Gallop, after famous creationist Duane Gish who favored this strategy).

   Plausibility in Science-Based Medicine
Steven Novella
Link
A question that arises often when discussing the optimal role of science in medicine is the precise role of plausibility, or prior probability. This is, in fact, the central concept that separates (for practical if not philosophical reasons) science-based medicine (SBM) from evidence-based medicine (EBM).
The concept featured prominently in the debate between myself and Dr. Katz at the recent Yale symposium that Kimball Atwood recently discussed. Dr. Katz’s treatment of the topic was fairly typical of CAM proponents, and consisted of a number of straw man derived from a false dichotomy, which I will describe in detail below.
I also recently received (I think by coincidence) the following question from an interested SBM reader:
What would Science Based Medicine do if H. pylori was not known, but a study showed that antibiotics given to patients with stomach ulcers eliminated symptoms? I assume that SBM wouldn’t dismiss it outright saying that it couldn’t possibly be helping because antibiotics don’t reduce stomach acid. I assume a SBM approach would do further studies trying to discover why antibiotics work. But, in the meantime, would a SBM practitioner refuse to give antibiotics to patients because he doesn’t have a scientific explanation as to why it works?
This is the exact type of scenario raised by David Katz during our discussion. He claimed that strict adherence to the principles of SBM would deprive patients of effective treatments, simply because we did not understand how they work. This is a pernicious straw man that significantly misconstrues the nature of plausibility and its relationship to the practice of medicine.
Plausibility
Plausibility is essentially an application of existing basic and clinical science to a new hypothesis, to give us an idea of how likely it is to be true. We are not starting from scratch with each new question – which would foolishly ignore over a century of hard-won biological and medical knowledge. Considering plausibility helps us to interpret the clinical literature, and also to establish research priorities. But plausibility is not the ultimate arbiter of clinical truth – it must be put into context with clinical evidence, just as clinical evidence must be put into the context of scientific plausibility.
One common mistake when considering plausibility is to reduce it to a false dichotomy – a claim is either plausible (which is falsely equated to scientists understanding its precise mechanism of action), or implausible (which is falsely equated to not knowing the mechanism of action). Rather, at least three broad categories need to be considered with regard to plausibility.
The first category are those treatments with a known mechanism or mechanisms of action that should, according to our existing models, produce a certain clinical effect. For example, we know that beta blockers bind and inhibit beta receptors in the heart and on blood vessels and thereby reduce cardiac output and dilate arteries which lowers blood pressure. It is therefore very plausible that beta blockers would have a protective effect against syndromes that result in an excess of catecholamine (adrenalin) production, since catecholamines bind and activate beta receptors.
We may also add to this category treatments for which there is anecdotal or preliminary evidence for efficacy – clinical plausibility.
There is still a range of plausibility within this category, but in such cases there is at least some reason to think that a treatment should work. The core principle of EBM, however, is that even in such situations we still need clinical studies looking at net health effects to show that plausible treatments are safe and effective – plausibility is not enough.
The next broad category is not implausible, but neutral or unknown with respect to plausibility. For such treatments we have no particular reason to think that they should work, but no reason to suspect that they do not or cannot work either. This category would include any pharmacological substance with an unknown mechanism of action, or mechanisms that are not known to interact with the disease or symptom being treated. There is no reason to think that beta blockers would improve memory in dementia, but this is not inherently implausible. Beta blockers are drugs, and may have other effects that have not yet been discovered.
This category applies to the question above – if we had reliable clinical data that showed antibiotics worked for ulcers, even though we had no idea how, we would still accept and even use this treatment (assuming the clinical data was sufficient). This of course would then lead to further investigation – is the beneficial effect due to a pharmacological property of the antibiotic not related to its antibiotic effects, or are some ulcers perhaps caused by or exacerbated by a bacterial infection.
Dr. Katz argued that SBM eliminates the possibility of serendipity – discovering new treatments by accident through clinical observations. But this is simply not true (one of his straw men) – SBM considers all the evidence, clinical and basic science. If clinical evidence is solid, that is enough, and often in the history of medicine lead to discoveries about mechanism and biology.
The arrow of research can go both ways – understanding plausibility can lead to new treatments, but discovering new treatments can lead to discoveries about biology and mechanism. The two play off each other.
But there is a third category in the plausibility spectrum – treatments that are inherently implausible. These are treatments that not only lack a known mechanism of action, they violate basic laws of science. Homeopathy violates the law of mass action (a basic principle of chemistry), the laws of thermodynamics (extreme dilutions maintaining the chemical “memory” of other substances), and all of our notions of bioavailability and pharmacokinetics.
Homeopaths therefore substitute any notion of chemical activity with a vague claim about “energy” – but this just puts homeopathy in the category of energy medicine, which is just as implausible. Invoking an unknown fundamental energy of the universe is not a trivial assumption. Centuries of study have failed to discover such an energy, and our models of biology and physiology have made such notions unnecessary, resulting in the discarding of “life energy” as a scientific idea over a century ago.
Essentially any claim that is the functional equivalent to saying “it’s magic” and would, by necessity, require the rewriting not only of our medical texts, but physics, chemistry, and biology, can reasonably be considered, not just unknown, but implausible.
Dr. Katz and others would like us to believe that this category does not exist, based upon the premise that we do no yet understand enough science to make such judgments. They often invoke vague references to quantum mechanics or the counter-intuitive nature of subatomic physics or cosmology to make their point. But this is an anti-intellectual and unscientific approach – it denies existing knowledge.
The alternative (often another false dichotomy and straw man) is not that we know everything – no one claims that. But not knowing everything is not the functional equivalent of knowing nothing. We do know stuff, and it is folly to deny the accumulated knowledge of the last few centuries of organized science.
Having said that – even the most implausible claim can still prove itself with sufficient clinical evidence. If homeopathy actually worked, it could be demonstrated through repeated rigorous clinical studies (something which has never happened). Admittedly, the bar for such evidence would be as high as the prior implausibility of the claim – which is very high – but if it really worked, that bar of evidence should theoretically be reachable. In that very hypothetical situation, the results would be extremely intriguing – clearly there would be something fundamental missing from our understanding of the relevant areas of science – a situation that often results in Nobel prizes.
Conclusion
SBM is ultimately about achieving the optimal relationship between science and the practice of medicine. SBM requires considering all the science, in its proper context, and does not follow any simplistic algorithm as is often suggested by critics. We look at what is known and what is unknown, at basic science and clinical evidence, and we put it all together, making an individual judgment for each individual claim.
We also are students of history – what claims have prospered or failed in the past, and what patterns predict ultimate success or failure? One pattern that should be obvious is that of highly implausible claims (not merely unknown) that can only produce weak and preliminary evidence, where more rigorous evidence tends to be negative, and positive evidence cannot be replicated, followed by special pleading by proponents. That is the pattern of a treatment that does not work.
We have seen this pattern with treatments that are now not controversial in their failure – phrenology, radioactive tonics, animal magnetism, and Abram’s dynomyzer (turned out to be a black box with loose non-functioning parts). We see the same pattern with homeopathy, therapeutic touch, energy medicine, and acupuncture.
We also see the same pattern for highly implausible (what some would consider pathological) fringe sciences outside of medicine – ESP research, ghost hunting, free energy, and cryptozoology, for example. There are also non-controversial historical examples, such as N-rays.
Of course, those who have not learned the lessons of history are doomed to repeat it.

Why Scientific Plausibility Matters
Why plausibility must play a central role in scientific medicine.
Steven Novella on May 31, 2023
Link

Applying the scientific method to medicine revolutionized the profession. While this was a long process, there was an inflection point, at least in the US, when the Flexner Report was published in 1910. The report was the basis for aligning US medical education with the standards already more widespread in Europe, and lead to a purge of pseudoscience and poor academic quality from American medical education.
But it seems that medical science, while thriving by some measures, is having some serious problems. The pessimistic view is that medical science is broken, but I think the full picture is far more complicated. We have been trying to unravel and make sense of this situation here at SBM. That is the core of our project – what is wrong with science and medicine and how do we fix it? At the center of the SBM approach is to emphasize the role of scientific plausibility in evaluating all medical claims, something that evidence-based medicine (EBM) deemphasizes, and in some applications even eliminates. But why is plausibility so important?
I think this is part of the natural evolution of scientific medicine. At first, when rigorous scientific methods were applied to medical questions, the questions were big – what causes infections, what are nutritional factors? Big questions had big impacts, and large effect sizes. We essentially have picked all the low hanging scientific fruit. As medical science progressed, the questions have become more subtle, nuanced, and complex. We are chasing smaller and smaller effect sizes. This requires greater and greater scientific rigor, and we have not always kept up.
Another factor is that all aspects of scientific institutions have become more expert at doing science, but this cuts both ways. This expertise can make certain players better at gaming the system. Working the system, in turn, has been driven by specific perverse incentives – academics want funding and to get promoted, institutions want greater reputations, journals want to maximize their impact factor, companies want to maximize their profit.
What does “gaming the system” mean in science? It amounts to getting good at creating the impression of a positive scientific result when there is no actual phenomenon. One aspect of this is P-hacking, tweaking protocols to manufacture statistically significant results, but there are many others. How do we know this is a problem in medicine? One way is because of the “replication crisis”. While there is still debate about exactly how big a problem this is, it is true that not all positive findings in science can be replicated.
The failure to replicate, however, is a feature of science, not a bug. Replication is partly how we determine if a phenomenon is real, which by definition means that some findings won’t replicate. If all positive findings replicated, then we wouldn’t need replication. But the devil is in the details. The real questions are – is the rate of failure to replicate too high, and what is happening with the results in the meantime. The real troubling statistics are that high impact journals are more likely to public studies that don’t replicate, and that such studies are more likely to be cited – even after they fail to replicate. The information ecosystem within the scientific medical literature favors new and exciting findings, not those which are actually true. And this does affect how we practice medicine.
An interesting angle to the replication problem is that people are generally good at predicting which studies are likely to replicate. Even lay people, when presented with a description and the evidence, we able to predict later replication with 67% accuracy, while experts averaged about 72%. Lay people were not technically analyzing methodology – they were basing their judgements on plausibility. Results that seem implausible have a lower probability of replicating – because the results likely do not reflect reality. Plausibility matters, and even non-experts can smell the difference.
Another line of evidence that there is a vulnerability within the institutions of science and medicine is the body of research into highly implausible claims. Interpreting this evidence is a key difference between SBM advocates and proponents of so-called alternative medicine (or more generally, scientific skeptics and believers in the paranormal). Essentially there is a conflict between plausibility and outcome. Homeopathy is a relatively clean example. When I look at studies that purport to show that homeopathy works, I conclude that the methodology of those studies must be flawed (likely due to P-hacking) and would predict that those studies would not replicate. Believers, however, characterize that position as “closed minded” and say that the fault is in our scientific knowledge, not the methods of those studies.
Which interpretation is more likely to be true? That depends on plausibility and methodology. The less plausible the claim, the more likely it is that a positive result is erroneous and won’t replicate. The more rigorous the methodology, the more seriously we should take the results. These two factors have to work in tandem. No matter how rigorous the methodology seems, if the plausibility is low enough it still becomes more likely that the results are spurious.
This can come down to just math. Scientific results are often statistical, and the p-value gives us an estimate of the probability we would be seeing this data if the hypothesis were not true. What are the odds, for example, that we would be seeing a positive result from homeopathic treatments if homeopathy does not work for the indication being studied? Even an extremely impressive p-value of 0.001 means the odds are one in a thousand (assuming perfect methodology and execution of the study, which is not always a deserved assumption). But we have to compare that to the odds that our current understanding of the laws of physics, biology, physiology, and biochemistry are all sufficiently wrong to allow for homeopathy to work.
And this is where the disagreements typically exist – how do we estimate that probability? I find that proponents often casually toss aside the prior findings of science and are comfortable assuming profoundly deep levels of ignorance about how the universe works. Or they simply brush aside concerns over plausibility as being “closed minded” or practicing “scientism”. In fact, we are just practicing science. When it comes to homeopathy, I would place the probability that current scientific knowledge is sufficiently wrong to allow for the possibility that homeopathy works at millions or billions to one – orders of magnitude less likely than even a p-value of 0.001 being a statistical quirk. (I have summarized the reasons why multiple times, but here is a good one.)
The same is true of claims such as ESP or astrology. Acupuncture is more plausible, because at least something physical is happening, but if you define acupuncture as requiring the existence of acupuncture points, the plausibility again plummets.
But it is important to understand that an SBM or skeptical analysis of these claims does not end with an assessment of plausibility. Despite the low plausibility, we still undertake a technical analysis of the literature. Not surprisingly, in every case (so far – but I am always willing to keep looking) the claims with extremely low plausibility all show that same patterns of methodology and results. There is always some potentially fatal flaw with the methodology, and the findings never convincingly replicate. What we never see is a quality of evidence above the threshold where rejecting the null hypothesis is reasonable. This is likely not a coincidence (that low probability claims also tend to have low quality or flawed evidence).
What we never see is rigorous studies with significant positive effects, reasonable effects sizes, and consistent replication. What we do see is gaming of the system, with a range of sophistication, but generally unconvincing evidence, and cries of being unfair for even considering plausibility.

Responding to Dr. Vinay Prasad’s “dunking on a 7′ hoop” criticism of SBM
Last week, Dr. Vinay Prasad decided to resurrect his misguided criticism from over a year ago likening SBM-style criticisms of alternative medicine to “dunking on a 7′ hoop”. Why he would decide to renew his attack on combating science disinformation in the middle of a pandemic that has killed close to 300,000 Americans alone we don’t know, but Drs. Novella and Gorski could not let such an off-base attack go unanswered.
David Gorski on December 13, 2020
Link

Last week, a prominent academic oncologist named Dr. Vinay Prasad published an article in MedPage Today entitled “Applying Skepticism to Medical Skepticism“. (I know, I know, the whole “skeptical of skeptics” or “skeptical of skepticism” bit is a tired, unoriginal, and unimaginative trope that never ceases to irritate skeptics and supporters of science-based medicine when we see it.) In a clear shot across the bow of this blog, though, the eminent Dr. Prasad’s article was subtitled: “Debunking should focus on areas most in need — which might not be homeopathy”. Further making it clear that he was referring to us and those like us here at SBM who devote a lot of time to deconstructing the bad science and pseudoscience behind a lot of claims of alternative medicine and the antivaccine movement, Dr. Prasad doubled down on an analogy he’d made before, that of “dunking on a 7′ hoop”:
A year ago, I got myself in trouble on Twitter. I tweeted a video of a man dunking on a basketball hoop. The young man took a big dribble and leapt into the air. The camera panned outward. He appeared to soar, defying gravity like Lebron James. He raised the ball up high and brought it down in a monster jam on the hoop. The camera continued to pan outward, and it became clear — the hoop was not the regulation 10-foot rim, but something much smaller. The man appeared to be dunking on a 7-foot hoop. My tweet went something like this “Dunking on alternative medicine is like dunking on a 7-foot hoop; let’s raise it back up to 10 feet.”
Basically, the whole thesis behind his post and analogy is that “dunking” on alternative medicine is so easy (like LeBron James dunking on a 7′ hoop) that it’s a waste of a physician’s intellectual firepower to bother with it. Instead, argues Dr. Prasad, we skeptics who’ve been “debunking” homeopathy should do what he does and devote our intellectual firepower to bad science in medicine and to conventional medical interventions for which the scientific evidence is not robust. In his post, he basically seems to think that all we at SBM and other skeptics do is “dunk on” homeopathy, jade eggs, and other Goop-like nonsense marketed to people with, as British comedians David Mitchell and Robert Webb once put it describing homeopathy, “more money than sense.”
Of course, the first time that Dr. Prasad made his argument on Twitter and in an episode of his podcast, which was late last year, a number of skeptics, myself included, responded by pointing out that we do, in fact, also do exactly what he said we should be doing and that dissecting clinical trials of various alternative medicine requires very much the same sort of skillset that dissecting any other clinical trial does, plus the additional skillset of knowing the alternative medicine claims and why they are pseudoscience. Sadly, Dr. Prasad revealed the disingenuousness behind his attacks by portraying such arguments as a tacit admission that we “medical skeptics” actually agreed with him that what he does is so much more important than what we do. Even so, ultimately he deleted a whole lot of those Tweets, for whatever reason, and blocked a lot of us on Twitter.
Why Dr. Prasad decided now to double down on his original analogy, who knows? What Steve Novella and I do know is that Dr. Prasad’s argument and dismissal of what skeptics do with respect to unscientific claims are particularly irresponsible during a pandemic in which unscientific claims about COVID-19, rooted in conspiracy theories, have arguably led to many tens of thousands more deaths by provoking uncertainty, distrust, and outright fear and hostility over masks, social distancing, and other public health interventions, and, now that the Pfizer COVID-19 vaccine has been granted emergency use authorization by the FDA and the Moderna vaccine is likely soon to follow suit, about vaccines. That’s why we wrote rebuttals on Friday:
    • “Skeptical of Skepticism regarding Medical Skepticism“, in which Steve Novella basically makes fun of the “skeptical of skepticism” trope in the title of Dr. Prasad’s article before explaining how Dr. Prasad goes wrong and observes that, when it comes to what skeptics do (and even how pervasive alternative medicine and antivaccine claims are), Dr. Prasad does not know what he is talking about.
    • “Dunking on a 7′ hoop, or: Refuting Dr. Vinay Prasad’s attack on medical skepticism“, in which a certain alter-ego does the same, only with more—shall we say?—insolence.
As editor of SBM and someone who has spent the last two decades dealing with harmful pseudoscientific varieties of medicine, such as cancer quackery and the antivaccine movement, all I can say is that it is very depressing to see that, 24 years after Carl Sagan’s death, academics like Dr. Prasad still don’t “get it,” to the point of still viewing science communication about things that they don’t consider intellectually challenging with such contempt and disdain. It’s particularly frustrating how, during a pandemic that has killed close to 300,000 Americans and still rapidly counting, Dr. Prasad can’t see the connection between “debunking” homeopathy and combatting the sort of conspiracy theory-based pseudoscience, misinformation, and disinformation that is contributing to making that death toll continue to climb. Dr. Prasad and we skeptics should be natural allies, as our areas of focus are complementary. It’s a shame he doesn’t see it that way.

   Please Don’t Define “Complementary and Alternative Health Practices”!
William M. London 
Link
Since I have a master’s and doctoral degree in health education and since I’m a professor in a department of public health with an undergraduate curriculum that includes substantial attention to health education, I participate in the email discussion group of HEDIR, the Health Education Directory. On August 16th, I received a message to the discussion group from the American Association for Health Education inviting participants to complete an online survey from the Joint Committee on Health Education and Promotion Terminology with results to be analyzed at the Committee’s meeting in September 2011.
The survey items include various terms used by health educators, the currently approved terminology, and three choices followed by a type-in box:
    • This term should remain as defined
    • This term should remain in the report but modified in definition
    • This term is no longer commonly used in health education/health promotion literature
If modify, please provide the suggested wording and reference for that definition if you are citing it from a specific source.
For one of the terms, my preferred response would have be have been a fourth choice that was not offered: The term is commonly used in health education/health promotion and elsewhere, but it should not be used because its use only serves to distort our thought processes and promote quackery.
Here is the term along with the definition presented in the survey:
Complementary and Alternative Health Practices: These practices generally include natural substances, physical manipulations, and self-care modalities. These approaches often incorporate aspects of interventions derived from traditional practices. The approach in Western societies has been to select specific approaches from these systems and apply them to health maintenance, health enhancement, or disease management. Such approaches can be used to compliment[sic] conventional allopathic care (complementary therapy), or as an alternative to conventional approaches (alternative therapy). Many of these complementary and alternative approaches have not been validated through experiential research, but those that have, such as acupuncture for pain, are being integrated into conventional health practices (integrative medicine).
And here are my objections to the term and to the definition given:
“Complementary and alternative health practices” is marketing doublespeak that conceals how promoting (via advertising, publicity, direct selling, word-of-mouth, etc.) non-validated or invalidated practices is unethical. When a practice is science-based, it is simply part of good healthcare or health promotion. “Complementary and alternative” jargon is never necessary to describe validated practices in health promotion or health care delivery. Science-based uses of natural substances, physical manipulations, and self-care modalities are all part of regular medicine.
Science-based natural products medicine is called pharmacognosy. Labels like “complementary and alternative” are used to give the impression of legitimacy, not to pharmacognosy, but to superstitious and often ecologically destructive uses of natural products such as herbalism (particularly paraherbalism), gruesome extractions of bile from living bears, shark cartilage, and rhinoceros horns.
Physical manipulations with a rational basis such as many of those included in personal exercise programs and physical therapy do not require euphemistic labels such as “complementary” or “alternative.” However, the labels “complementary” or “alternative” are often applied to give the appearance of legitimacy to superstition-based or pseudoscience-based physical manipulation treatments such as those used in chiropractic. Many chiropractors falsely claim that the spine requires periodic maintenance “adjustments” of health compromising “subluxations” that only chiropractors can supposedly detect. Such adjustments don’t complement anything else and they aren’t a viable alternative for health promotion or disease prevention.
Many of the manipulations promoted as “complementary” or “alternative” are actually non-physical; they are rooted in vitalism, which is defined as: “a doctrine that the functions of a living organism are due to a vital principle distinct from physicochemical forces” or “a doctrine that the processes of life are not explicable by the laws of physics and chemistry alone….” Different health cults have different names for the supposed vital principle. In anthroposophy, the names are the divine element in nature, astral body, formative force, or either body. In Ayurvedic medicine, it’s prana. In chiropractic, it’s innate intelligence. In Reichian psychotherapy, it’s orgone energy. In homeopathy, it’s vital energy. In naturopathy, it’s vis medicatrix naturae. In Traditional Chinese Medicine and acupuncture, it’s chi or qi or ki.
The vital principle was popularized in Star Wars as “the force.” But in the real universe, nothing like “the force” is reliably detectable and there are no Jedi-like masters who can manipulate anything akin to it for healing or any other purposes. In the movie Austin Powers: The Spy Who Shagged Me, the vital principle was called mojo. The idea that some people have skills as mojo detectors or mojo manipulators is as absurd as the movie, but “there are some ideas so absurd that only an intellectual could believe them.”
Self-care modalities have been promoted as “complementary” and “alternative,” but what useful distinction is there to be made between supposed “complementary” or “alternative” self-care modalities and those that don’t qualify and are therefore implicitly non-complementary or non-alternative? If the distinction is to be based on validation for safety and efficacy, why introduce euphemistic language like “complementary” or “alternative”? Categories such as validated, non-validated, and invalidated should suffice to give consumers useful information for deciding what modalities of self-care are worth trying out.
It’s true that many tradition-based practices are promoted as “complementary” or “alternative.” Scholars who attempt to advance “complementary and alternative medicine” often like to emphasize traditional systems of care and ignore other practices marketed as “complementary and alternative.” Since it is often considered rude to be judgmental about traditions associated with particular cultures, fallacious appeals to traditional wisdom are useful in public relations. But numerous practices, products, and services marketed to consumers as “complementary” or “alternative” are promoted as “innovative,” “advanced,” “cutting edge,” “modern,” “scientific,” and the like, not as tradition-based. Examples include such so-called complementary and alternative medicine approaches (sCAMs) as metabolic therapy, chelation therapy, oxygenation treatments, insulin potentiation therapy, clinical ecology, anti-aging medicine, attachment therapy, various other mental health therapies, antineoplastons, cellular therapy, and syncrometers & zappers.
Referring to the selection of specific approaches from traditional systems in Western societies as “complementary” or “alternative” implies an East-West dichotomy that is simply false. Tradition-based systems and supposed whole-system care are not uniquely Eastern. Is it only in Western societies that approaches from traditional systems get used separately from whole-system care?
Medical anthropologists, medical sociologists, educated laypersons, health educators, and even physicians often make the mistake of describing standard medical practices of today and recent decades as “conventional allopathic care.” Allopathy is a term coined by Samuel Hahnemann (formulator of homeopathic treatment principles) as a label for medical practices of his day that were based upon ancient Greek humoral theory of disease such as bleeding and purging and blistering to manipulate the four so-called body humors: blood, phlegm, black bile, and yellow bile. As medicine became more science-based, it discarded treatment based upon the convention of manipulating body humors and progressed by developing healthcare consistent with progress in biological and physical sciences. Nevertheless, approaches to healthcare based upon humoral theory—what Hahnemann called allopathy—persist today in parts of India, Pakistan and elsewhere as Unani medicine, which, ironically, the World Health Organization recognizes as a type of “CAM.” Unani is an Arabic adjective meaning Greek.
Since modern medicine makes progress by relying on science, it is iconoclastic—the antithesis of conventional. By contrast, the real allopathy practiced today as Unani medicine is bound to its ancient conventions. Like much of what gets promoted as “complementary” and “alternative,” Unani medicine reflects conventional wisdom of healing traditions rather than the rigor of scientific testing and the iconoclasm of scientific discovery.
I have previously explained that calling an approach to healthcare “complementary” implies that it adds to the outcome when combined with some other treatment and that calling an approach to healthcare “alternative” implies that it can be successfully used in lieu of some other approach. However, this is misleading labeling. Simply calling an approach “complementary” doesn’t mean it actually complements anything else and calling an approach “alternative” doesn’t make it a viable alternative. The jargon “complementary and alternative” serves to distract attention away from questions of utility based upon scientific merit.
Professor Richard Dawkins has explained: “Either it is true that a medicine works or it isn’t. It cannot be false in the ordinary sense but true in some ‘alternative’ sense.”
Drs. John E. Dodes and Marvin Schissel put it this way: “Erythromycin is an alternative to penicillin, but a pogo stick is not an alternative to an automobile.”
Drs. Marcia Angell and Jerome Kassirer wrote: “There cannot be two kinds of medicine—conventional and alternative. There is only medicine that has been adequately tested and medicine that has not, medicine that works and medicine that may or may not work.”
Dr. George Lundberg explains it this way:
There is no “alternative medicine.” There is only medicine:
    • Medicine that has been tested and found to be safe and effective. Use it; pay for it.
    • And, medicine that has been tested and found to be unsafe or ineffective. Don’t use it; don’t pay for it.
    • And, medicine for which there is some plausible reason to believe that it might be safe and effective. Test it and then place it into one of the other two categories.
Although many people believe that acupuncture for pain is medicine that fits Dr. Lundberg’s first category, the weight of evidence places it in the second category, especially considering the lack of a plausible rationale for acupuncture as a therapy. Few, if any, health practices that have been promoted as “complementary” and “alternative” also belong in Dr. Lundberg’s first category. More than ten years of research funding by the National Center for Complementary and Alternative Medicine has failed to contribute to medical progress.
The term integrative medicine is superfluous and should not be used by responsible health professionals. Palliative care and adjunctive care are meaningful and useful terms for efforts to provide rational modalities of humane care, comfort, and support addressing the diverse needs of patients. The term “integrative medicine” adds nothing to describe approaches strongly supported by scientific evidence, but serves as an income-generating mechanism for attracting patients to seemingly special modalities that typically lack support beyond cherry-picked evidence or tradition. The term “integrative medicine” is not needed to offer science-based psychological approaches for managing health problems, but it does help in marketing when you are offering modalities based on vitalism. “Integrative medicine” represents branding, not a meaningful medical specialty. It projects a misleading image of academic seriousness that serves only to obscure its hype and help secure funding for clinical research of dubious need.
Terms such as “alternative,” “complementary,” and “integrative” have become popular euphemisms for non-validated and invalidated approaches to health enhancement—especially approaches with farfetched rationales. The use of such euphemisms facilitates quackery: the promotion of health products, services, or practices of questionable safety, effectiveness, or validity for an intended purpose. Today quackery is a far less popular term than the euphemisms. In some circles, it is politically incorrect to refer to quackery. But if we cannot refer to quackery as quackery, we can expect it to persist as a neglected public health scandal. I suggest that there are better alternatives to using currently popular euphemisms of alt-speak.

Statistics Done Wrong, And How To Do Better
Harriet Hall on August 2, 2016
Link
Statistics is hard, often counterintuitive, and burdened with esoteric mathematical equations. Statistics classes can be boring and demanding; students might be tempted to call it “Sadistics.” Good statistics are essential to good research; unfortunately many scientists and even some statisticians are doing statistics wrong. Statistician Alex Reinhart has written a helpful book, Statistics Done Wrong: The Woefully Complete Guide, that every researcher and everyone who reads research would benefit from reading. The book contains a few graphs but is blissfully equation-free. It doesn’t teach how to calculate anything; it explains blunders in recent research and how to avoid them.
Inadequate education and self-deception
Most of us have little or no formal education in statistics and have picked up some knowledge in a haphazard fashion as we went along. Reinhart offers some discouraging facts. He says a doctor who takes one introductory statistics course would only be able to understand about a fifth of the articles in The New England Journal of Medicine. On a test of statistical methods commonly used in medicine, medical residents averaged less than 50% correct, medical school faculty averaged less than 75% correct, and even the experts who designed the study goofed: one question offered only a choice of four incorrect definitions.
There are plenty of examples of people deliberately lying with statistics, but that’s not what this book is about. It is about researchers who have fooled themselves by making errors they didn’t realize they were making. He cites Hanlon’s razor: “never attribute to malice that which is adequately explained by incompetence.” He says even conclusions based on properly done statistics can’t always be trusted, because it is trivially easy to “torture the data until it confesses.”
Misconceptions about the p value
A p value of less than .05 is considered statistically significant by convention. It represents the probability of getting the results they got if the null hypothesis is true (that there is no difference between the two groups). Steve Novella has written about problems with using p-values before. Reinhart warns:
Remember, a p value is not a measure of how right you are or how important a difference is. Instead, think of it as a measure of surprise [emphasis added]. If you assume your medication is ineffective and there is no reason other than luck for the two groups to differ, then the smaller the p value, the more surprising and lucky your results are – or your assumption is wrong, and the medication truly works.
He explains confidence intervals and effect size and shows that they are much more meaningful than significance testing. As an example, five large meta-analyses set out to answer the question “do statins reduce mortality in patients with no history of cardiovascular disease?” Three meta-analyses answered yes, two answered no (based on finding or not finding significant differences at the p <0.05 level). People reading that would think the evidence was contradictory and inconclusive. But he presents a graph of confidence intervals showing that all five studies actually showed an effect of around 0.9: 10% fewer patients on statin drugs died. In the graph, only two of the studies’ confidence intervals overlapped the “no difference” line, and only by a very small amount. Looking at it this way, it is obvious that all of the meta-analyses actually showed that the answer was yes, statins do reduce mortality in patients with no history of cardiovascular disease.

Common errors
Studies have shown that when women live together their menstrual cycles become synchronized. Reinhart eviscerates that flawed research and shows that menstrual synchrony is (still!) a myth. He mentions the infamous dead salmon MRI study and explains the errors that are commonly committed in brain imaging studies. He covers double-dipping in the data, the problem of multiple comparisons, truth inflation, the base rate fallacy, pseudoreplication, underpowered studies, circular analysis, regression to the mean, correlation is not causation, the loss of information when a continuous variable is arbitrarily dichotomized into two categories, regression models, missing data, degrees of freedom, methods to avoid bias, descriptions of methods that are too incomplete to allow replication by others, the need for open sharing of data, the file drawer and publication bias problems, the Reproducibility Project, reporting bias, poor compliance with registering clinical trials and reporting all their findings, and poor compliance with guidelines like CONSORT.
He shows why stopping rules must be defined in advance: it is tempting to stop the study at a point where the data favor your hypothesis, but further data collection might reverse the findings. He explains Simpson’s paradox, where an apparent trend in data is caused by a confounding variable and can be eliminated or even reversed by splitting the data into meaningful groups. He advocates not just double-blind, but triple-blind experiments, where the statisticians analyzing the data don’t know which group it came from. He shows the pitfalls of using computer programs to analyze statistics rather than consulting live statisticians. He reveals that errors are common in published studies: 38% of the papers in the journal Nature had typos and calculation errors in their p values, and reviews have found examples of mistakes like misclassified data, erroneous duplication of data, or even using the wrong dataset entirely.
When a new genetic test promised to tailor chemotherapy to the individual patient’s specific variant of cancer, analysis by statisticians showed the research to be such a mess that no conclusions could be drawn.
Possible solutions
He mentions several proposed statistical reforms (abandoning p values, switching to confidence intervals, switching to new Bayesian methods) but thinks they all have merits and doesn’t pick one to advocate. He calls for better statistics education with new evidence-based teaching methods aimed at identifying and overcoming students’ misconceptions and holes in their knowledge. He advocates reforms in scientific publishing. He gives specific guidelines for reading a scientific paper. In every chapter, he points out common errors and then gives concrete advice about how researchers can avoid them.
An excellent book
This book is well written, short (129 pages), enjoyable to read, and easy to understand. It is well-organized, divided conveniently into short chapters, and is enhanced by the examples of research done wrong that illustrate each of its points. It covers the subject thoroughly yet concisely, and provides copious references for readers who want to know more. Rather than a guide to statistics, it is really more of a guide to critical thinking about pitfalls in the use of statistics. I highly recommend it. Whether you are interested in research or statistics, want to know what to look for when reading a research paper, or just want to improve your critical thinking skills, you will find it of value. Even if you are an expert in statistics, you might think it deserves a place on your bookshelf.

Fun With Statistics
Harriet Hall on December 4, 2012
Link
Statistics is the essential foundation for science-based medicine. Unfortunately, it’s a confusing subject that invites errors and misunderstandings. We non-statisticians could all benefit from learning more about statistics as well as trying to get a better understanding of just how much we don’t know. Most of us are not going to read a statistics textbook, but the book Dicing with Death: Chance, Risk, and Health by Stephen Senn is an excellent place to start or continue our education. Statistics can be misused to lie with numbers, but when used properly it is the indispensable discipline that allows scientists:
…to translate information into knowledge. It tells us how to evaluate evidence, how to design experiments, how to turn data into decisions, how much credence should be given to whom to what and why, how to reckon chances and when to take them.
Senn covers the whole field of statistics, including Bayesian vs. frequentist approaches, significance tests, life tables, survival analysis, the problematic but still useful meta-analysis, prior probability, likelihood, coefficients of correlation, the generalizability of results, multivariate analysis, ethics, equipoise, and a multitude of other useful topics. He includes biographical notes about the often rather curious statisticians who developed the discipline. And while he includes some mathematics out of necessity, he helpfully stars the more technical sections and chapters so they can be skipped by readers who find mathematics painful. The book is full of examples from real-life medical applications, and it is funny enough to hold the reader’s interest.\\
What a Difference a Word Makes
Statistics (and probabilities) are frequently misunderstood, even by many scientists. Even what looks simple can turn out to be complicated and counter-intuitive. Senn revisits an old question. If a man has 2 children and at least one of them is a boy, how likely is it that the other is a girl? Most people reason that there are only 2 possibilities, boy or girl, both equally likely, so there is a probability of 1 in 2, or 50%, that the other child is a girl. That’s wrong. In fact, there is a probability of 2 in 3: the other child is twice as likely to be a girl as a boy. The 50% answer is only true if you change the question slightly from “one of them is a boy” to “the firstborn is a boy.” If this doesn’t make sense to you, you really need to read the book.
Does Medical Research Discriminate Against Women?
He devotes a whole chapter to that question and answers it with a convincing NO. The Women’s Caucus complained that a lot of studies had only male subjects, that women were under-represented, that what we were learning about treating men might not apply equally to women. The perception that women were being neglected was not supported by the evidence; if anything, women have actually been over-represented. Nevertheless, they persuaded congress to pass a law requiring the director of the NIH to ensure that trials be designed to examine whether the variables being studied affect women or members of minority subgroups. Senn explains how confounding factors like sex and race were already being dealt with adequately and why a strict enforcement of the new policy would be disastrous for research, requiring far greater numbers of subjects and greater expense. Fortunately, researchers have managed to continue their previous rational and appropriate practices while making a few token placatory noises to the grant-making bodies.
Applications Beyond Medicine
Medicine is the only profession that employs randomized controlled trials to evaluate the effects of its actions and systematically encourages publications about its errors. Evidence-based medicine is a good thing, and there’s no reason the methods of statistics couldn’t be used to develop similarly evidence-based approaches to educational, managerial, economic, and social policies. He describes one intriguing application: the process of religious conversion has been studied using infectious disease modeling.
Statistics in the Courtroom
The Dow Corning company manufactured silicone breast implants until statistical innumeracy allowed a flurry of successful lawsuits to bankrupt the company. Senn quotes the definition of dice from The Devil’s Dictionary:
Small polka-dotted cubes of ivory, constructed like a lawyer to lie on any side, but commonly on the wrong side.
He also includes this discouraging quote from Marcia Angell:
I am occasionally asked by lawyers why the New England Journal of Medicine does not publish studies “on the other side”, a concept that has no meaning in medical research… Yet science in the courtroom, no matter how inadequate, has great impact on people’s lives and fortunes.
Statistics, MMR Vaccine, and Autism
In the last chapter he revisits the Wakefield fiasco, showing how misunderstandings of statistical principles have led to outbreaks of vaccine-preventable diseases. Statistics has an important role in determining rational public policies to protect the population, and through its sub-science of decision analysis has much to say about the ethical and economic aspects of vaccination.
Entertainment Value
Senn has a droll wit and an endearing fondness for puns. Who would have thought that a book on a dry subject like statistics could be so entertaining and even laugh-out-loud funny? He says the post hoc fallacy is so obvious that “almost any human being who is not a journalist can understand it.” Since it regularly gets by the average journalist, he renames it the “post hoc, passed hack” fallacy. He says that John Donne’s “no man is an island” is true, but if you capitalize Man it becomes “false, as anyone cognizant with the geography of the Irish Sea will surely aver.” In case geography-challenged readers don’t get this, he reluctantly explains it in the endnotes. He says such punctuation would be a capital offense. His puns extend to this tour de force:
Poisson was attracted to magnetism, stretched his mind on the theory of elasticity, starred in astronomy and generally counted as one of the foremost mathematicians of his day.
His sources are wide-ranging and eclectic. He quotes Guernsey McPearson’s definition of a meta-analyst as one who thinks that if manure is piled high enough it will smell like roses. His quotes from Sinclair Lewis’s Arrowsmith inspired me to re-read that book (it’s available online and it’s just as pertinent today as it was in 1925.) One of his quotes from The Phantom Tollbooth is particularly applicable to some areas of CAM:
I only treat illnesses that don’t exist: that way if I can’t cure then, there’s no harm done.
Conclusion
The book reinforced my conviction that I don’t know enough about statistics and never will, so I will have to continue to rely on the experts. I confess that some of the material was over my head. I couldn’t follow the formula for Russian roulette showing that the probability of death on the 5th attempt is 0.066 compared to 0.1 on the first attempt (no, the initial probability is not the 1 in 6 that you would suppose, since the weight of the bullet makes the spinning cylinder more likely to stop with the bullet at the bottom). I couldn’t follow his explanation of two different mathematical approaches to coin toss probabilities. But I do know more about statistics now than I did when I started reading. I learned much of value and was well entertained in the process.
I didn’t need any convincing, but I think even the most skeptical reader would come away convinced that:
the calculation of chances and consequences and the comprehension of contingencies are crucial to science and indeed all rational human activity.

   Prior Probability: The Dirty Little Secret of “Evidence-Based Alternative Medicine”
Kimball Atwood 
Link
This is actually the second entry in this series;† the first was Part V of the Homeopathy and Evidence-Based Medicine series, which began the discussion of why Evidence-Based Medicine (EBM) is not up to the task of evaluating highly implausible claims. That discussion made the point that EBM favors equivocal clinical trial data over basic science, even if the latter is both firmly established and refutes the clinical claim. It suggested that this failure in calculus is not an indictment of EBM’s originators, but rather was an understandable lapse on their part: it never occurred to them, even as recently as 1990, that EBM would soon be asked to judge contests pitting low powered, bias-prone clinical investigations and reviews against facts of nature elucidated by voluminous and rigorous experimentation. Thus although EBM correctly recognizes that basic science is an insufficient basis for determining the safety and effectiveness of a new medical treatment, it overlooks its necessary place in that exercise.
This entry develops the argument in a more formal way. In so doing it advocates a solution to the problem that has been offered by several others, but so far without real success: the adoption of Bayesian inference for evaluating clinical trial data.
Many readers will recognize that the term “prior probability” comes from Bayesian statistical analysis. They may correctly conclude that at least part of the reason to prefer Bayesian over “frequentist” statistical evaluations of clinical trials—which have been dominant throughout the careers of every physician now alive—is that the former require considering evidence external to the trial in question. That, of course, is what we should be doing in any case, but it helps to have a formal reminder. Bayes’ Theorem shows how our existing view (the prior probability) of the truth of a matter can be altered by new experimental data. Prior probability must be estimated from all existing evidence: basic science, previous clinical trials, funding sources, investigators’ identities and histories, and other factors. How conclusions based on such evidence might be altered by new data is illustrated by this statement of Bayes’ Theorem:
Where:
P stands for probability;
A is the hypothesis in question;
| stands for “given”; and
B is the data generated by the trial at hand.
Thus P(A|B), the probability of the hypothesis given the data (also called the “posterior probability”), is proportional to P(B|A), the probability of the data given the hypothesis, and also to P(A), the “prior probability” of the hypothesis. P(A|B) is inversely proportional to P(B), the probability of producing the data.
We might not know P(B), but it is a constant. Thus on the right side of the equation we can direct our attention to the terms in the numerator, which predict certain things: if the prior probability of a hypothesis is high, it will not require much in the way of confirming data to reassure us of that opinion. If the prior probability of a hypothesis is small, it will require a large amount of credible, confirming data to convince us to take it seriously. If the prior probability is exceedingly small, it will require a massive influx of confirming data to convince us to take it seriously (yes, extraordinary claims really do require extraordinary evidence). The simplest result, albeit one that many find discomfiting, is found if P(A) approaches zero: no amount of “confirming data”—especially of the error-prone sort generated by a clinical trial—should convince us to accept the hypothesis.
It turns out that the last assertion, although undeniably true, is not necessary to make the case for the superiority of Bayesian statistics in clinical research. I didn’t know that until I had read the following two articles:
Goodman SN. Toward evidence-based medical statistics. 1: The P value fallacy. Ann Intern Med. 1999;130:995-1004.
Goodman SN. Toward evidence-based medical statistics. 2: The Bayes factor. Ann Intern Med. 1999;130:1005-13.
Dr. Goodman makes the arguments for Bayesian inference in far more compelling ways than I can. Thus after a brief introduction I’ll quote him liberally—and ask his forgiveness in advance for whatever embarrassing inaccuracies I will be or may have already been guilty of spouting.
Dr. Goodman observes that it is the subjective nature of prior probabilities (“measuring ‘belief’ ”) that explains why clinical trial literature has shied away from Bayesian statistics, instead favoring the familiar “frequentist statistics” with its P values, confidence intervals, and hypothesis tests: tools that are widely assumed to provide objective measures of evidence for hypotheses by looking exclusively at data from trials. Those tools don’t provide such objective measures, however, nor can they. As any scientist and most physicians know, it is foolish to evaluate trial results without considering external knowledge. What most don’t know, however, is that “frequentist statistics” are irrational tools for the job that they have been assigned to do, and that they include methods that are not compatible even with each other. These points are introduced in the abstract of the first article cited above:
An important problem exists in the interpretation of modern medical research data: Biological understanding and previous research play little formal role in the interpretation of quantitative results. This phenomenon is manifest in the discussion sections of research articles and ultimately can affect the reliability of conclusions. The standard statistical approach has created this situation by promoting the illusion that conclusions can be produced with certain “error rates,” without consideration of information from outside the experiment. This statistical approach, the key components of which are P values and hypothesis tests, is widely perceived as a mathematically coherent approach to inference. There is little appreciation in the medical community that the methodology is an amalgam of incompatible elements, whose utility for scientific inference has been the subject of intense debate among statisticians for almost 70 years. This article introduces some of the key elements of that debate and traces the appeal and adverse impact of this methodology to the P value fallacy, the mistaken idea that a single number can capture both the long-run outcomes of an experiment and the evidential meaning of a single result. This argument is made as a prelude to the suggestion that another measure of evidence should be used—the Bayes factor, which properly separates issues of long-run behavior from evidential strength and allows the integration of background knowledge with statistical findings.”
The “intense debate” that Dr. Goodman refers to is over a problem central to science: that of “inductive” vs. “deductive” reasoning. As many will recall from college philosophy courses, “inductive” reasoning uses observations to generate hypotheses: if the first 10,000 swans one sees are white, then a reasonable (tentative) hypothesis is that all swans are white. That is the way science, including clinical trials, usually works. The obvious problem with it is that it can’t be conclusive: the 10,001st swan might be black. “Deductive” reasoning begins with a principle and makes predictions: if at least some swans are white, then the next one we see has a probability > zero of being white. Deductive reasoning is logically sound, but has obvious limitations as a tool for learning about nature.
It turns out that “frequentist statistics” not only lacks a formal way to consider external evidence, but is inappropriate for evaluating clinical trials for a more fundamental reason: it applies only to deductive inference. Thus
…when physicians are presented with a single-sentence summary of a study that produced a surprising result with P = 0.05, the overwhelming majority will confidently state that there is a 95% or greater chance that the null hypothesis is incorrect. This is an understandable but categorically wrong interpretation because the P value is calculated on the assumption that the null hypothesis is true. It cannot, therefore, be a direct measure of the probability that the null hypothesis is false. This logical error reinforces the mistaken notion that the data alone can tell us the probability that a hypothesis is true.” (emphasis added)
On the other hand,
Determining which underlying truth is most likely on the basis of the data is a problem in inverse probability, or inductive inference, that was solved quantitatively more than 200 years ago by the Reverend Thomas Bayes.”
Dr. Goodman explains this expertly, but for me it still required a few reads before it began to sink in. (Please, dear reader, if you hope to see Evidence-Based Medicine become synonymous with Science-Based Medicine, tackle these articles.) The final sentence in the abstract above introduces the second point about Bayesian statistics that I had not previously appreciated, which follows from its application to inductive inference: it is not necessary to dwell on Prior Probability estimates to appreciate the superiority of the Bayesian method. Another term in the theorem, known as the Bayes Factor, is calculated entirely from objective data but is a more useful and accurate “measure of evidence” than the familiar “P value.” The Bayes Factor is illustrated in this statement of Bayes’ Theorem (from Dr. Goodman’s second article):
(Prior Odds of Null Hypothesis) X (Bayes Factor) = Posterior Odds of Null Hypothesis
Where Bayes factor = Prob(Data, given the null hypothesis)
Prob(Data, given the alternative hypothesis)
The abstract of Goodman’s second article continues the discussion (emphasis added):
Bayesian inference is usually presented as a method for determining how scientific belief should be modified by data. Although Bayesian methodology has been one of the most active areas of statistical development in the past 20 years, medical researchers have been reluctant to embrace what they perceive as a subjective approach to data analysis. It is little understood that Bayesian methods have a data-based core, which can be used as a calculus of evidence. This core is the Bayes factor, which in its simplest form is also called a likelihood ratio. The minimum Bayes factor is objective and can be used in lieu of the P value as a measure of the evidential strength. Unlike P values, Bayes factors have a sound theoretical foundation and an interpretation that allows their use in both inference and decision making. Bayes factors show that P values greatly overstate the evidence against the null hypothesis. Most important, Bayes factors require the addition of background knowledge to be transformed into inferences—probabilities that a given conclusion is right or wrong. They make the distinction clear between experimental evidence and inferential conclusions while providing a framework in which to combine prior with current evidence.”
At this point I must stop, but let me suggest a fun project: pick a “CAM” study of an implausible hypothesis such as homeopathy, “distant healing,” or whatever, that has been evaluated by “frequentist statistics” and purports to demonstrate an effect “significant at P=.04” or so. Now, using your new knowledge of inductive inference, re-evaluate the data using a few points from a range of prior odds of the null hypothesis being true, say 8 to 1—99,999 to 1 (which are far more favorable to homeopathy, for example, than established knowledge warrants). You needn’t even make calculations: both Goodman and Ioannidis provide tables and nomograms that can help you estimate the answers.
Here is a short additional bibliography, including a couple of shameless pitches for offerings by two of your humble bloggers:
1. Browner W, Newman T. Are all significant P values created equal? The analogy between diagnostic tests and clinical research. JAMA. 1987;257:2459-63.
2. Stalker DF. Evidence and Alternative Medicine. The Mt. Sinai J of Med.1995 62;2:132-143
3. Brophy JM, Joseph L. Placing trials in context using Bayesian analysis. GUSTO revisited by Reverend Bayes. JAMA. 1995;273:871-5.
4. Lilford R, Braunholtz D. The statistical basis of public policy: a paradigm shift is overdue. BMJ. 1996;313:603-7.
5. Freedman L. Bayesian statistical methods [Editorial]. BMJ. 1996;313:569-70.
6. Sampson WI, Atwood KC IV. Propagation of the absurd: demarcation of the absurd revisited. MJA 2005; 183 (11/12): 580-581
7. Atwood KC IV. Prior Probability: the dirty little secret of “Evidence-Based Alternative Medicine.” Talk given at the 11th European Skeptics Congress. London, September 5-7, 2003

   Prior Probability: the Dirty Little Secret of “Evidence-Based Alternative Medicine”—Continued
Kimball Atwood
Link
This is an addendum to my previous entry on Bayesian statistics for clinical research.† After that posting, a few comments made it clear that I needed to add some words about estimating prior probabilities of therapeutic hypotheses. This is a huge topic that I will discuss briefly. In that, happily, I am abetted by my own ignorance. Thus I apologize in advance for simplistic or incomplete explanations. Also, when I mention misconceptions about either Bayesian or “frequentist” statistics, I am not doing so with particular readers in mind, even if certain comments may have triggered my thinking. I am quite willing to give readers credit for more insight into these issues than might be apparent from my own comments, which reflect common, initial difficulties in digesting the differences between the two inferential approaches. Those include my own difficulties, after years of assuming that the “frequentist” approach was both comprehensive and rational—while I had only a cursory understanding of it. That, I imagine, placed me well within two standard deviations of the mean level of statistical knowledge held by physicians in general.
Opinions are Inescapable
First, a couple of general observations. As Steven Goodman and other Bayesian advocates have argued, we do not now avoid subjective beliefs regarding clinical hypotheses, nor can we. There is not a legitimate choice between making subjective estimates of prior probabilities and relying exclusively on “objective” data, notwithstanding the wishes of those who zealously cling to the “frequentist” school. The inescapable fact is that there is no logical resolution to the age-old epistemological dilemma of induction vs. deduction. “Frequentist” statistical tools, designed for deductive inference, do not address the question of how data from a particular experiment affect the probability of a hypothesis being correct—an inductive exercise. 
The simplest illustration of the problem, as discussed by Goodman and others, is the “P-value fallacy“: while most physicians and many biomedical researchers think that a “P” of 0.05 for a clinical trial means that there is only a 5% chance that the null hypothesis is true, that is not the case. Here is what “P=0.05″ actually means: if many similar trials are performed testing the same novel hypothesis, and if the null hypothesis is true, then it (the null) will be falsely rejected in 5% of those trials. For any single trial, it doesn’t tell us much.
“Non-Informative Priors”: Bayes Factors are the new P-values
Turning to the matter at hand, it’s important to reiterate that the usefulness of Bayesian analysis does not depend exclusively on unique or known prior probabilities, or even on ranges that a group of experts might agree upon. The power of Bayes’ Theorem is to show how data from an investigation alter any prior probability to generate a new, “posterior probability.” Thus one class of prior probabilities, dubbed “non-informative” or “reference priors,” is arbitrary. As discussed in Goodman’s second article (pp. 1006-8), their usefulness stems from a term in Bayes’ Theorem that is entirely objective: the Bayes Factor. Recall that the Bayes Factor is a “likelihood ratio,” which in its simplest form compares how well the data reflect the best supported hypothesis to how well the data reflect the null hypothesis. This means that the Bayes Factor can be the useful “measure of evidence” for a single trial that most of us had previously assumed, incorrectly, the “P-value” to be. Goodman advocates this use and develops an “exchange rate” between the two values (p. 1007).
This can be illustrated by an example. A few years ago a report concluded: ”we found that supplementary, remote, blinded, intercessory prayer produced a measurable improvement in the medical outcomes of critically ill patients.” The effect was said to be demonstrated by lower “CCU course scores” in the prayed-for group compared to the controls, which the authors deemed “statistically significant” because ”P=.04.”
Although there were numerous flaws in the study, this discussion will be restricted to the P-value claim. The previous couple of paragraphs will have already alerted the reader to the fallacy of the “P=.04″ assertion, but let’s put the cards on the table:
The authors appeared to believe, and seemed to expect readers to believe, that their data showed that there was only a 4% chance of a “null” effect.
Referring to Goodman’s assumptions for “exchange rates” between P values and Bayes Factors, P=.04 can be shown to be equivalent to a “minimum” Bayes factor (i.e., the one most consistent with the data) of slightly less than 0.15 (Table 2, p. 1008).
Thus picking an arbitrary, “non-informative” prior probability of 50% (even odds) would yield a posterior probability (by Bayes Theorem) of the “null” effect of about 12%: far greater than the ≤5% that we have been taught to think of as “statistically significant” evidence of a treatment effect.
Using the same table, it can be seen that to be truly 96% confident of a non-null effect after the study, the prior probability of “intercessory prayer”—people praying from a distance for patients they do not know and who are unaware of it—would have to have been about 70%. (That is, the prior probability of the null hypothesis would have to have been about 30%).
By starting with a “neutral” prior, using Bayesian statistics in this case demonstrates that the data did not show what the investigators concluded. Alternatively, there needn’t have been a unique prior probability estimate, or the investigators might have been asked to offer their own estimates—even if those might have seemed unrealistically “enthusiastic”—which could then have been compared to “skeptical” priors offered by others. Both of those terms appear in Bayesian literature as examples of arbitrary priors—or so I gather. Whatever the estimates, they would have been available for others to scrutinize. 
Extrapolating to more reasonable prior probability realms (see below), it would have required far more dramatic findings to reach a posterior probability of ≥95% of a non-null effect. If we also impose varying estimates of bias, the posterior probabilities of a non-null effect become lower, maybe vastly lower.
Informative Priors
Estimating “reasonable prior probability realms”—which offers the fullest use of Bayes’ Theorem for clinical trials and health care policy questions—is a topic of major interest in the field. Rather than address it myself (I’m not competent to do so), I’ll make a brief introduction and then refer readers to more informative sources. First, although there is always a subjective element, there are systematic means to estimate “priors,” and these add considerable rigor to the exercise. Second, such priors are expected to be “transparent,” i.e., the bases for their derivation must be clearly stated. This is quite different from the way subjective opinion is typically introduced now, as illustrated on p. 1002 of Goodman’s first article. Third, priors (of all types) are usually given as distributions, not as discrete numbers. Fourth, they are expected to be derived from all pertinent information. This is from the non-quantitative, online Primer on Bayesian Statistics in Health Economics and Outcomes Research (O’Hagan & Luce): 
The ‘Evidence’
Prior information should be based on sound evidence and reasoned judgements. A good way to think of this is to parody a familiar quotation: the prior distribution should be ‘the evidence, the whole evidence and nothing but the evidence’:
‘the evidence’ – genuine information legitimately interpreted;
‘the whole evidence’ – not omitting relevant information (preferably a consensus that pools the knowledge of a range of experts);
‘nothing but the evidence’ – not contaminated by bias or prejudice.
A more sophisticated, quantitative treatment can be found here. Peruse the table of contents, especially chapter 5.
Most Hypotheses are Wrong…
There is at least one broad theme regarding estimating prior probabilities. It is virtually self-evident that most hypotheses are wrong. This follows from two premises: 1. the number of possible hypotheses is limited only by human imagination; 2. most people’s imaginations are not informed by a sophisticated appreciation for nature or science. Even those who are exceptions to the second premise are humbled by the elusiveness of fruitful insights. Consider the words of Nobel laureate Peter Medawar, one of the great immunologists of the 20th century:
It is a layman’s conclusion that in science we caper from pinnacle to pinnacle of achievement and that we exercise a method which preserves us from error. Indeed we do not; our way of going about things takes it for granted that we guess less often right than wrong…
It is likely that the multitude of less gifted scientists will guess right even less often. John Ioannidis, whose recent paper “Why most published research findings are false” is a favorite among your SBM bloggers, asserts in his article that “the majority of modern biomedical research is operating in areas with very low pre- and post-study probability for true findings.” (Hold the phone! Steven Goodman, our other hero of Bayesian analysis, disagreed with Ioannidis, although not entirely; Ioannidis replied here). 
According to FDA Consumer,
More often than many scientists care to admit, researchers just have to give up when a drug is poorly absorbed, is unsafe, or simply doesn’t work. The organization Pharmaceutical Research and Manufacturers of America estimates that only 5 in 5,000 compounds that enter preclinical testing make it to human testing, and only 1 of those 5 may be safe and effective enough to reach pharmacy shelves.
Even granting a degree of self-serving exaggeration by “BigPharm,” it is clear that most proposed, biologically plausible drugs never pan out. 
…Hence Most Priors are Low (but you still have to estimate each on its own terms)
Add to all of that the ridiculous certitudes of the hopelessly naive, and it is clear that there is a surfeit of guessing wrong about how things work. Does that mean that most proposals for trials have a prior probability of, say, less than 0.5? You bet it does (much less, in most cases), although that fact does not properly figure into the estimate of any single proposal.
Later: some words about other useful comments regarding the “Galileo gambit” and more.

   Prior Probability: the Dirty Little Secret of “Evidence-Based Alternative Medicine”—Continued Again
Kimball Atwood
Link
After the previous posting on the Bayesian approach to clinical trial data, several new comments made it clear to me that more needed to be said. This posting addresses those comments and adds a few more observations regarding the unfortunate consequences of EBM’s neglect of prior probability as it applies to “complementary and alternative medicine” (“CAM”).†
The “Galileo Gambit” and the Statistics Gambit
Reader durvit wrote:
A very interesting example, for a number of people, might be estimating the prior probability for Marshall and Warren’s early work on Helicobacter pylori and its impact on gastroduodenal management. I frequently have Marshall quoted to me as a variation on the Galileo gambit, so establishing whether he and Warren would have been helped or hindered by Bayesian techniques would be useful.
This suggestion raises a couple of issues. First, the “Galileo gambit” regarding Marshall and Warren’s discovery is a straw man (as durvit seems to have surmised).
The second issue is one that I might have overlooked if durvit hadn’t called attention to it: not all experiments require statistical analyses. Occasionally there are results that are so clear-cut, and that fit with prior knowledge so well, that a few independent replications make a convincing case for the hypothesis. Although it may come as a surprise to some, that was the case for H. pylori and peptic ulcer disease.
In extreme summary, for M and W there were three, or maybe four steps: 1. finding “unidentified curved bacilli” in enough specimens of diseased gastric epithelium (at first gastrititis, later ulcer) to become convinced that the bacteria were real and possibly pathogenic, and thus to pursue them (reported in 1983); 2. culturing and thereby isolating and characterizing the new bacterium (a difficult job because of its fastidiousness, but with a little luck it finally worked—reported in 1984); 3. replications of their bacteriologic findings by huge numbers of researchers all over the world within a couple of years of their report of the first successful culture, and the common finding that the organism was found in diseased tissues, thus lending credence to M and W’s early suspicion of pathogenicity; 4. therapeutic trials of eradicating H. pylori to observe its effect on peptic ulcer disease.
For the first 3 steps, either the finding (a new species of bacteria) was there or it wasn’t. Growing it in culture was a challenge, but that consisted of recognizing a few new wrinkles in standard bacteriology: what media it liked, what other environmental requirements it had (5% oxygen), how long it took to grow (a long time, which is why M and W almost gave up). No need for P values or prior probabilities there. Step 4, therapeutic trials, might have needed a little statistical help except that the results were so dramatic and the context so ready: eradicating H. pylori with antibiotics reduced the recurrence rates of healed peptic ulcers from 50-90% to about 10%, a result that was consistently found in several, independent trials. A slam-dunk: no statistical analyses necessary.
By “context,” I mean that by the time of the therapeutic trials, most people in the field were ready to accept the pathogenic role of H. pylori because of its predictable association with diseased tissue. The H. pylori hypothesis, as reader daedalus suspected, became plausible as soon as the organism was recovered and cultured. Ulcers are inflammatory processes, and inflammatory processes are commonly caused by bacteria. The “acid environment” of the stomach is child’s play for the adaptive capacities of prokaryotes, as every self-respecting microbiologist (if not every MD) by that time knew. The only legitimate skepticism, before M and W cultured the organism, would have been based on previous, failed attempts at culturing it. But those attempts had been made in the early 1950s, prior to a more sophisticated appreciation of the wide diversity of prokaryotic morphologies, staining requirements for microscopy, and “lifestyles.” Mycoplasma, for example, a common cause of atypical pneumonia, was thought to be a virus until the early 1960s.
The Marshall and Warren story is thus one in which the choice of statistical schools was beside the point.
Bayesian Analysis and Sequential Studies
Reader Mark raised an important point with this statement:
Suppose, hypothetically, that some new altmed treatment came out that sounded utterly ludicrous, but then was found to be effective in study after study, and replicated under controlled conditions by skeptical researchers. You would simply look at that research and say “well, I estimate the prior P of this treatment working as epsilon, since it sounds so stupid, therefore these studies don’t prove anything.”
I hadn’t discussed this previously because I was emphasizing the superiority of the Bayesian approach to individual studies, but Bayesian reasoning would not support such a conclusion. In any series of studies consistently finding that the novel hypothesis explains the data better than the null hypothesis does, the probability of the former must increase sequentially because of those studies. According to O’Hagan and Luce:
Today’s posterior is tomorrow’s prior.
The paradigm is about learning, and we can always learn more. When we acquire more data, Bayes’ theorem tells us how to update our knowledge to synthesize the new data. The old posterior contains all that we know before seeing the new data, and so becomes the new prior distribution. Bayes’ theorem synthesizes this with the new data to give the new posterior. And on it goes…Bayesian methods are ideal for sequential trials!
Notice that this also deals with hypotheses whose prior probabilities are initially deemed higher than the eventual reality, and thus shows how a series of trials will inevitably result in a convergence of disparate, initial probability estimates to some single distribution that everyone, from initial skeptic to initial enthusiast, must agree upon (please don’t challenge me on this point; I’m aware that it assumes that all observers will remain true to the mathematics, will agree that the data are credible, and maybe some other things that I’m not thinking of. I’ll say no more of that).
We Frequently (and inescapably) rely on Opinions but we Pretend Otherwise
A reiteration: clinical research is inductive and Bayes’ Theorem (simply proved, by the way) is the way to figure out how new data alter the probability of a hypothesis being true. There is no escaping it. Opinion already figures heavily in current clinical research and practice, but P-values and confidence intervals encourage researchers to pretend that it does not. One of the benefits of Bayesian statistics is to force opinion out into the open where it belongs. Prior probabilities, if stated, must be formally estimated (usually in the form of a distribution) and therefore justified. Readers may or may not agree, and the inevitable debates contribute to the progress of the field and let onlookers know where the debaters stand.
That happens even now, but it’s done largely post hoc, is opaque rather than transparent, and lacks both quantitation and a rational link to the data. Dr. Goodman gives examples in his “P-value” article. Covert opinion also figures in medical practice, obviously, but without a Bayesian perspective it is likely to be tentative and incomplete. Regarding the option to use highly implausible treatments themselves, for example, most physicians and patients seem to ignore “positive evidence” that must seem a bit wacky to them.
That’s a reasonable, if underdeveloped and non-quantitative estimate of prior probability that has stood modern physicians and patients in good stead. Many of those same people, however, when asked to weigh in on ”Complementary and Alternative Medicine” (“CAM”) policy issues—e.g., research projects and funding, medical school curricula, or state endorsements of “CAM” practices—will bend to the weight of the “evidence,” EBM-style, not realizing that their personal misgivings are well-founded. That can lead to unfortunate consequences.
Failing to consider Prior Probability leads to Unethical Human Studies
An example, I suspect, was the regrettable decision of two academic pediatricians, one of whom is a nationally-recognized expert in pediatric infectious disease, to become co-investigators in an uncontrolled trial of homeopathic “remedies” for acute otitis media (AOM) in 24 children, ages 8 mos-77 mos. The treating investigators were homeopaths. The report provided evidence that 16 of the children had persistent or recurrent symptoms, lasting from several days to 4 weeks after diagnosis. Nevertheless, no child was prescribed analgesics or anti-pyretics, and only one child was eventually prescribed an antibiotic by an investigator (another in an emergency room). There is no evidence that the investigators evaluated any of the subjects for complications of AOM, nor did the two academic pediatricians “interact with any research subjects.” Similar examples are not hard to find. [1,2]
Funny thing about EBM’s tenacious hold on medical academics: a few years ago, when I first noticed the report just described, I ran it by a friend who is the chief of pediatrics at a Boston area hospital and a well-known academic pediatrician in his own right. After I explained the tenets of homeopathy, he agreed that it is highly implausible. At that point I expected him to agree that the trial had been unethical. Instead he queried, “but isn’t its being unproven just the reason that they should be allowed to study it?” There was no convincing him.
Prior Probability, Misleading Language, and Mr. Magoo
Even many researchers who should know better do not. I am happy that statistician R. Barker Bausell, who worked on “CAM” projects for several years, has finally seemed to realize that they are bogus. But what took him so long? Bausell doesn’t mention Bayes or prior probability in his book, although he seems to understand the P-value fallacy (pp. 172-4). In over 300 pages he limits his discussion of the “theory” of several “CAM” claims to a couple of paragraphs each, and in most cases he misses the point. Or perhaps his explanations merely conform to curious, recent anti-scientific admonitions to be “respectful” or “balanced” when discussing “CAM.” [3] This quotation is from Bausell’s two paragraphs on how homeopathy is “hypothesized to work” (p. 261):
There is no known physiological principle that can explain why “similars” or “like cures like” should work…[and] nothing else in nature that we know of becomes more potent as it is diluted. So the mechanism of action for this therapy is simply beyond the province of science…This is not to say, however, that the evaluation of these drugs’ clinical effectiveness is outside the ken of science, since also by definition they are especially suitable for placebo-controlled trials.
In their defense, homeopaths place much more emphasis upon developing a positive, caring relationship with their patients than do most conventional practitioners, and they probably spend more time with them as well, trying to understand both the patient and the genesis of his or her illness.” [etc.]
Coming from an active (or formerly active) “CAM” researcher, that is about as good as it gets. Nevertheless, I hope that readers who followed the entirety of the “Homeopathy and EBM” blog will have recognized the misleading language and erroneous content of those statements, however unintended on Bausell’s part. There is a big difference between the narrow truth of “no physiological principle that can explain why ‘similars’…should work” and the whole truth of the matter, to wit: the basis for Hahnemann’s claim of “similars” is categorically wrong. Similarly, “nothing else in nature that we know of becomes more potent as it is diluted” is understated in the extreme: to be true the claim would likely violate the 1st (energy is conserved), and certainly violate the 2nd (entropy increases) laws of the universe. To say that “the mechanism of action for this therapy is simply beyond the province of science” is to offer an epistemological safe haven for fools, akin to “evolution is only a theory.” It is also untrue, as Bausell surely knows, because there are quite plausible explanations for the “mechanism of action for this therapy.”
To suggest that science should evaluate homeopathy by doing placebo-controlled trials, thereby side-stepping established knowledge, is how EBM (and Bausell) got sucked into the “CAM” mess in the first place. Finally, crediting homeopaths for “trying to understand both the patient and the genesis of his or her illness” is plainly fatuous, referring as it must to their baseless and incoherent elicitations of “symptoms,” and the similarly daft processes of “provings” and prescribing—suggesting that Bausell, who is not a physician, has either been duped by homeopaths’ advertising pitches or, for some reason, feels that he must be conciliatory toward them.
Bausell praises the Cochrane Collaboration for offering the most “unbiased assessments of what works and what doesn’t work” and for “not being susceptible to accusations of bias toward ‘CAM.’ “ (p. 202) Yet almost all Cochrane “CAM” reviews avoid stating that the method under review “doesn’t work,” whereas in most cases there is overwhelming external evidence that this is the case. Rather, the reviews typically advise that the ”evidence”—in the EBM sense—doesn’t yet support recommending the use of the method, and “more research” is necessary. The coy suggestion of future promise and the refusal to consider all the evidence betray a bias in favor of “CAM” claims—claims that inevitably wither under the stark light of what we already know about nature.
All of this is reminiscent of a point made by Dr. Goodman, regarding the unfortunate legacy of “frequentist” statistics in medical research:
This method thus facilitated a subtle change in the balance of medical authority from those with knowledge of the biological basis of medicine toward those with knowledge of quantitative methods, or toward the quantitative results alone, as though the numbers somehow spoke for themselves.
On the jacket of Bausell’s book is an approving blurb by Edzard Ernst, “the first Professor of Complementary Medicine in the United Kingdom.” Dr. Ernst writes that he and Bausell “both have researched CAM rigorously for many years; we both were unable to show through our work that much of CAM works convincingly; and we are both publishing a book about this experience.” If it’s true that Dr. Ernst has also arrived at the truth about “CAM,” again I might wonder why it took so long—except that I already know the answer. In responding to a well-argued plea for rational thinking about “CAM,” written in 2003 by our friend and colleague Cees Renckens, Ernst dismissed plausibility while making the spurious claim, refuted above, that if we had considered it we might never have allowed the bacterial cause of peptic ulcer disease to be demonstrated. His argument was a perfect illustration of the EBM-”frequentist” fallacy as it bears on “CAM” research:
In the context of EBM, a priori plausibility has become less and less important. The aim of EBM is to establish whether a treatment works, not how it works or how plausible it is that it may work. The main tool for finding out is the RCT. It is obvious that the principles of EBM and those of a priori plausibility can, at times, clash, and they often clash spectacularly in the realm of CAM.
Mr. Magoo-like, Dr. Ernst somehow managed to avoid stumbling upon the truth at the last possible moment. We are reminded of the emotional ejaculation of David Reilly that began this series, heralding his own unwitting brush with reality.
Perhaps I’m being too hard on Ernst and Bausell. It is somewhat understandable that they have been blinded by the limited EBM version of evidence, and it is to their credit that they have finally recognized “CAM” for what it is in spite of that handicap (if, indeed, they have). Dr. Ernst, in particular, stands alone among academic “CAM” enthusiasts in having managed to have one foot planted in the skeptical camp all along. He has been the world’s most prolific university-based reviewer of “CAM” claims over the past 15 years, and although he is spectacularly wrong to favor the “principles of EBM [over] a priori plausibility” when the two “clash spectacularly,” his reviews have been mostly rigorous, in the EBM sense. He is also the author of one of my favorite rebukes to those who think that “licensing” and “credentialing” of “CAM” practitioners is in the public interest:
Those who believe that regulation is a substitute for evidence will find that even the most meticulous regulation of nonsense must still result in nonsense.
The obvious truth of that statement, even if its “evidence” is of the anemic, EBM variety, stands in square opposition to the views of most of Ernst’s colleagues, including several with whom he has collaborated (do they talk?). The U.S. government could have saved its taxpayers much money and misinformation, and the NIH much embarrassment, simply by paying Dr. Ernst to do exactly what he’s been doing since 1993, rather than having created the Office of Alternative Medicine and later the NCCAM.
Dr. Ernst, unlike his colleagues at the Cochrane Collaboration, at the NCCAM, and elsewhere, has also never been afraid to say that a “CAM” claim doesn’t work. Examples can be found here, here, and here. This is especially impressive in light of the original funding for his Professorship, from the Maurice Laing Foundation, for “research into efficacy of complementary health treatments and their integration into general medicine.” Placing the cart of “integration” before the horse of “research into efficacy”—language that is common to the NCCAM and other proponent organizations—would seem to favor advocacy over skeptical inquiry, and is contrary to Dr. Ernst’s words quoted above. Interesting: Laing no longer includes Ernst or his department on its list of fundees. Instead it endows the University of Southampton, where George Lewith can apparently be counted on to remain a company man. Has Dr. Ernst, by being too rigorous in his “CAM” evaluations, worn out his welcome?
If so, he has my congrats. Then why can’t I just take “yes” for an answer from Ernst and Bausell? The reasons are several and have already been touched upon or discussed at some length. In the latter category are evidence not considered and its close relative, statistics erroneously applied. In the former are unethical human trials that will keep happening until EBM opens its eyes to all the evidence; “CAM” advocates boldly demanding “pluralism” in health care; public discourse filled with distortions of language, of science, and of the scientific basis of medicine; and much more. None of this needed to happen, even if there were politics and money behind it. It couldn’t have emerged from the eternal, marginal “woo” fringe if there hadn’t been well-meaning but misguided academics such as Ernst, Bausell, and many more welcoming it with open arms and open wallets. Most of them still do.
References
[1] Gurney S. Socially Harmful but Unapparent Effects of the NCCAM – Columbia University “Gonzalez” Protocol. Sci Rev Alt Med 7;2:74-77. (Fall-Winter 2004).
[2] Atwood KC, IV, Woeckner E, Baratz RS, Sampson WI. Why the NIH Trial to Assess Chelation Therapy (TACT) Should be Abandoned. Medscape Journal of Medicine. Accepted for publication.
[3] Sampson WI. On Being a Critic. Sci Rev Alt Med 2;1:4-5 (Spring/Summer 1998).

The Plausibility Problem
David Weinberg on July 20, 2012
Link
From the very outset, the founders of Science Based Medicine have have emphasized the importance of plausibility in the critical evaluation of scientific claims in medicine. What exactly does “plausibility” mean, and how should we apply it in science? My simple definition of plausibility would be “the likelihood that a premise is true.” The application in science is a little more complicated.
Consciously or unconsciously, we all consider plausibility in interpreting events in our lives. For example, if one of your coworkers showed up late for work and grumbled about a traffic jam, you would likely accept his story without question. If, instead, the same coworker attributed his tardiness to an alien abduction, you would not be so charitable. In each case, he has provided the same level of evidence: his anecdotal account. You are likely to accept one story and reject the other because of a perceived difference in the plausibility. The skeptic’s mantra “Extraordinary Claims Require Extraordinary Evidence” expresses this concept in a qualitative way.
Evidence-based medicine has traditionally ignored plausibility when interpreting the evidence for a medical intervention. Science-based medicine, as envisioned by the creators of this blog, includes plausibility when making these judgements.
Since experiment research employs rigorous controls, and statistical criteria, you might assume that plausibility is not an issue, however, this is not entirely true. An article written by John Ioannidis entitled “Why Most Published Research Findings Are False” is cited frequently as a reference for the impact of plausibility on the interpretation of research results. This article enumerates numerous factor leading to erroneous research conclusions. Most of them have been dealt with on this blog at one time or another. To me, the most eye-opening aspect of the paper was a quantitive approach to the influence of plausibility in interpreting positive research findings. I was never taught this approach in medical school, or in any other venue. When it comes to implausible hypotheses, the traditional P-value can be very misleading.
As good as Ioannidis’ article is, it is not easy reading for the statistically or mathematically challenged. What I attempt to do in this post is to demonstrate the importance of plausibility in graphic format, without a lot of complex math. If you can grasp the concepts in this post, you will have an understanding that many researchers, and consumers of research, lack.
Research vs Reality
When scientists design research studies, they hope to obtain a result that leads them to a better understanding of reality. In medicine, studies are frequently undertaken to investigate the effectiveness of a treatment strategy.
The result of such a study can intersect with reality in 4 ways:
     1.  True Positive (TP): The treatment works and the study demonstrates this effectiveness
     2.  False Negative (FN): The treatment works, but the study fails to demonstrate this effectiveness
     3.  False Positive (FP): The treatment does not work, but the study shows it to be effective
     4.  True Negative (TN): The treatment does not work, and the study suggests it is not effective.
We can display the possible study results in a simple grid:
We would hope that all our results would be True Positives, or True Negatives. If that were the case, all research studies would be conclusive. Unfortunately, science is never that simple. There are ways to reduce, but not eliminate False Negative, and False Positive Results. As we go through the elements of a hypothesis, and study design, we will modify the table above in a way that the area of each of the 4 cells in the grid, is proportional to the likelihood of each particular outcome.
Lets start with a very basic study design. We are exploring a new treatment for disease X. We would consider the study positive, if the treatments is associated with improved outcomes at some pre-specified level of statistical significance.
Now lets look into the study details that can influence the relative likelihood of the 4 outcomes: TP, FN, FP, and TN.
Study Power
In designing a study, it is valuable to look at hypothetical results for a treatment that is effective, and for a treatment that is not effective.
We can first consider the possible results if, in reality, our treatment is effective. In this case we will obtain one of two possible results: True Positive or False Negative. These can be shown in a simple table.
There are design considerations that can increase the probability of obtaining a true positive result. The likelihood that a trial of an effective treatment will yield a TP result is called the study Power (also known as Sensitivity). Power can be expressed as a number between 0 and 1 (if you are more comfortable working in percentages, just multiply the Power times 100%). The higher the sensitivity, the more likely it is that we will confirm the effectiveness of our treatment. We can modify the table to include the Power of the study. The height of the TP and FP bars represent the relative likelihood of each of the 2 possible outcomes.
The Table above shows a study designed to find a TP result 80% of the time for an effective treatment. We would say that this study has a Power of 0.8 . In medical research 0.8 is considered quite a powerful study, and would usually require a large number of patients (large sample size).
Specificity and P values
In the previous section, we looked at possible outcomes for an effective treatment. Now we will look at possible outcomes for an ineffective treatment. If our treatment is ineffective, we will obtain one of 2 possible outcomes: True Negative or False Positive.
Researchers are careful to avoid false positive results. The parameter that defines the likelihood that a study will obtain a negative result if the drug is ineffective is known as the Specificity. Like power, specificity is expressed as a number between 0 and 1 (or between 0% and 100%). A .95 specific study will obtain a True Negative result for an ineffective treatment 95% of the time, and will obtain a False Positive result 5% of the time.
It is more common for researchers to talk about the false positive rate rather than the specificity. Researchers usually refer to the critical P-value rather than the Specificity. P-value and specificity convey exactly the same information, and can readily be derived from one-another. Specificity = 1 – P
When designing a study, the investigators select a critical P value. In order to accept the results of study favoring the study treatment, the P value must be at or below the predefined critical P value. For most studies in medicine, the critical P value is .05 (Specificity .95). This means that if they are studying an ineffective treatment, there is a 1 in 20 chance that they may declare this ineffective treatment effective. The lower the critical P value, the less likely a study is to find a false positive, but the tradeoff is that it becomes harder to find a true positive. The table below shows depicts a study with a critical p value of .05. The relative size of the boxes represents the likelihood of FP and TN results when a study premise is false.
Now lets combine the possibilities for effective and ineffective treatments. We can combine the 2 tables into one.
In the table above, the cells representing the TP, FN, FP, and TN outcomes are approximately equal size. Below is a more realistic representation of a study with a Power of .8, and a critical P of .05.
Now we have done our study, and analyzed the results. We have achieved our target P value of .05. What then is the likelihood that this positive study is a true positive? Since our P value is .05, we can be 95% sure that our study is correct, right? WRONG! Remember, the P value tells us something about the False Positives. The P value alone tells us nothing about the chances of a True Positive result! This is a very critical concept, and one that is poorly understood by many people.
Since our study power was .8, then we can be 80% sure that our result is true, right? Unfortunately no. The power tells us something about true positives relative to false negatives. We already know that our study is positive, but we want to know whether it is a True Positive or a False Positive.
Plausibility, Prior Probability, and Positive Predictive Value
We need a parameter that can help us distinguish between a true positive and a false positive result. Such a parameter is known as the Positive Predictive Value (PPV). The PPV considers both the True Positives and False Positives, and calculates the probability that our positive result is a True Positive. As with the other parameters, PPV is expressed as a number between 0 and 1, and calculated by the expression:
PPV = TP / (TP+FP)
But there is one more critical piece of information necessary to calculate the PPV of our study.
The Prior Probability is the likelihood, prior to beginning the study, that our premise is true. Prior Probability is a quantitative assessment of plausibility. Like sensitivity and specificity, prior probability can be expressed as a number between 0 and 1. A prior probability of 0 means that there is zero chance that the premise is true. A prior probability of 1 means that the truth of the premise is a certainty. In our table, we can display prior probability as the relative widths of the columns labeled “Treatment effective” and “Treatment ineffective”
For the study of our drug, lets assume the drug has a prior probability of .5, meaning that going into our study, there is a 50/50 chance that the drug is effective. This is probably a reasonable estimate for a drug with a valid preclinical foundation, and promising phase 2 data. In the table, a prior probability of 50% is reflected by the fact that the columns for “Treatment Effective” and “Treatment Ineffective” are of equal width.
Just to establish myself as Über Geek among SBM contributors I have created a Positive Predictive Value Calculator widget. The values are pre-populated for the problem above, but you can enter any values you wish (provided they are between 0 and 1). Remember: Sensitivity = Power; and Specificity = 1-P.
In truth we can rarely, if ever, calculate an accurate prior probability for any given premise. Even though we cannot come up with a truly accurate prior probability, it is instructive to explore the interaction between prior probability and positive predictive value.
When it comes to low prior plausibility, homeopathy is the go-to treatment ; a status it has earned for very good reason. For homeopathy to work, numerous principles of chemistry and physics would have to be wrong. Not just wrong, but spectacularly wrong. Established principles of physiology and pharmacology would also have to be wrong. Most of us would estimate the prior probability of any clinical claim of homeopathy for any indication as infinitesimal (irony intended).
Lets assign a very generous prior probability of .05 (5% likelihood that homeopathy is effective). If we factor this into a study with a power of .8 and a critical P value of .05, we come up with a table that looks like this. (note, not drawn to exact scale to preserve readability).
By adding prior probability we have changed the width of the columns to reflect the difference in probability of the various outcomes. Remember, the columns on the left represent the possible outcomes of an effective treatment, and the columns on the right represent possible outcomes of an ineffecive treatment. The area of each cell represents the relative likelihood of that result, given the parameters specified. A study with the parameters shown above, if it is fair, and free of bias has a low chance of finding any positive result. If the result is positive, however, the areas of the TP and FP cells are approximately the same size, meaning that they have a similar likelihood.
The positive predictive value of this study is reflected by the area of TP divided by the area of TP+FP. If we calculate the PPV this study we come up with a value of 0.46. So given a powerful study, and a VERY charitable prior probability for homeopathy, a positive clinical trial result for homeopathy has positive predictive value a little worse than a coin toss. If you lower the prior probability to more accurate .01 (1%) the positive predictive value becomes .14.
The Big Picture
Everything we have discussed can be summarized in the figure below. This depicts the relationship between Positive Predictive Value and Prior Probability for research studies of various powers. Notice that at the left side of the curve, where prior probabilities are very low, the positive predictive value approaches zero. Even for a powerful study design, when the prior plausibility is low, the positive predictive value is low. For less powerful studies, the positive predictive value is even weaker. We do not need agree on an exact value for the prior plausibility of homeopathy to see that any single positive clinical trial is virtually meaningless.
The horizontal axis represents the Prior Probability. The vertical axis represents Positive Predictive Value. The blue, green, and red curves represent results for studies with powers of .8, .5. and .25, respectively. All results assume a critical P value of .05.
The take-home message of the graph above is this: even for a well designed, powerful study, if the premise is highly unlikely, a positive result does not give us convincing evidence that the premise is true. For studies with weaker power, the results are even less persuasive.
So why do extraordinary claims require extraordinary evidence? Because for implausible claims, ordinary evidence is highly unreliable. A single positive study with a P value of .05 is ordinary evidence. For a very implausible hypothesis, a result of this sort is quite likely to be a false positive.

Plausibility bias? You say that as though that were a bad thing!
David Gorski on May 7, 2012
Link
On Friday, you might have noticed that Mark Crislip hinted at a foreshadowing of a blog post to come. This is that blog post. He knew it was coming because when I saw the article that inspired it, I sent an e-mail to my fellow bloggers marking out my territory like a dog peeing on every tree or protecting my newfound topic like a mother bear protecting her cubs. In other words, I was telling them all to back off. This article is mine.
Mine! Mine! Mine! I tell you!
My extreme territorial tendencies (even towards my friends and colleagues) notwithstanding on this issue aside, if you read Mark’s post (and if you didn’t go back and read it now—seriously, go now), you might also remember that he was discussing a “reality bias” in science-based medicine (SBM), a bias that we like to call prior plausibility. In brief, positive randomized clinical trials (RCTs) testing highly implausible treatments are far more likely to be false positives than RCTs testing more plausible treatments. That is the lesson that John Ioannidis has taught us and that I’ve written about multiple times before, as have other SBM bloggers, most prominently Kimball Atwood, although nearly all of us have chimed in at one time or another about this issue.
Apparently a homeopath disagrees and expressed his disagreement in an article published last week online in Medicine, Health Care, and Philosophy entitled Plausibility and evidence: the case of homeopathy. You’ll get an idea of what it is that affected us at SBM like the proverbial matador waving his cape in front of a bull by reading this brief passage from the abstract:
Prior disbelief in homeopathy is rooted in the perceived implausibility of any conceivable mechanism of action. Using the ‘crossword analogy’, we demonstrate that plausibility bias impedes assessment of the clinical evidence. Sweeping statements about the scientific impossibility of homeopathy are themselves unscientific: scientific statements must be precise and testable.
Scientific. You keep using that word. I do not think it means what you think it means. Of course, his being a homeopath is about as close to a guarantee as I can think of that a person doesn’t have the first clue what is and is not scientific. If he did, he wouldn’t be a homeopath. Still, this particular line of attack is often effective, whether yielded by a homeopath or other CAM apologist. After all, why not test these therapies in human beings and see if they work? What’s wrong with that? Isn’t it “close-minded” to claim that scientific considerations of prior plausibility consign homeopathy to the eternal dustbin of pseudoscience?
Not at all. There’s a difference between being open-minded and being so “open-minded” that your brains threaten to fall out. Guess which category homeopaths like Rutten fall into. But to hear them tell it, homeopathy is rejected because because we scientists have a “negative plausibility bias” towards it. At least, that’s what Rutten and some other homeopaths have been trying to convince us. This article seems to be an attempt to put some meat on the bones of their initial trial balloon of this argument published last summer, which Steve Novella duly deconstructed.
Before I dig in, however, I think it’s necessary for me to “confess” my bias and why I think it should be your bias too.
In which I confess my bias
Regular readers might have noticed that we write about homeopathy a lot on this blog. You might wonder why. Indeed, sometimes I myself wonder why. After all, if you want to come up with a list of the top three most ridiculous alternative medicine modalities with a large following, surely homeopathy will almost always be on the list, along with energy healing modalities (such as reiki) and a third nutty modality to be named later whose identity will be left for the reader for later given that there is likely to be some disagreement about it.
In any case, among highly implausible alternative medicine “healing systems,” homeopathy is at or near the top of the heap, reigning supreme. After all, given its twin pillars of “like cures like” and the law of infinitesimals, the former of which says that to relieve a set of symptoms you choose a remedy that causes those symptoms in healthy people and the latter of which says that those “like” remedies get stronger if they are highly diluted in serial steps—but only if they are vigorously shaken or “succussed” between each step. The first principle has no basis in physiology, pharmacology, biochemistry, or medicine (the claims of homeopaths to co-opt a real phenomenon known as hormesis notwithstanding), while the second principle so thoroughly violates the laws of chemistry and physics that, for it to be true huge swaths of these disciplines that have been well-established through hundreds of years of experimentation and observation would have to be not just wrong, but spectacularly wrong. One must concede that it’s possible that this latter principle might be true, but the odds that it is are about as infinitesimal as the amount of starting remedy in a 30 C homeopathic remedy. (That’s a 1 in 1060 chance, for those not familiar with homeopathy.) For all practical intents and purposes, the chances that homeopathy can work is zero. It is just water with its believer’s magical intent imagined into it.
So ridiculous is homeopathy that I sometimes feel that I and my fellow supporters of SBM are firing Howitzers at an ant when we take so much time and effort to explain why homeopathy is nonsense. On the other hand, it is homeopathy’s monumental lack of scientific plausibility that makes it a perfect teaching tool explaining the difference between science-based medicine (SBM) and evidence-based medicine (EBM). Specifically, because clinical trials have unavoidable shortcomings and biases, even at a p=0.05, which would imply only approximately a 5% chance that a given trial’s apparently positive results could be due to random chance alone. As John Ioannidis has taught us, in clinical trials as practiced in the real world, the chance is much higher that any given positive trial is a false positive. As explained in so much detail by Kimball Atwood, that also means that, the lower the prior plausibility of a remedy working, the higher the chance of false positive trials. This is exactly what we see in homeopathy, hence the panoply of homeopathy trials showing “positive” results in which the treatment group is barely different from the control and/or the results barely reach statistical significance. With something like homeopathy, which violates the laws of so many sciences, it is relatively easy to make the case that it takes a lot more than a few equivocal clinical trials to show that so much well-established science is wrong. Apparently positive clinical trials of homeopathy are measuring, in essence, the noise inherent in doing clinical trials.
Although most physicians and clinical investigators don’t think about it consciously they tend to have a bias for plausible hypotheses and treatments in evidence-based medicine and against implausible hypotheses. This bias is certainly not inherent in EBM, as we have described may times before. EBM, after all, relegates basic science considerations to the very bottom rung of its ladder of evidence, below even expert opinion. Clinical trial evidence and epidemiology are all, and, although EBM aficionados deny it, the way EBM is practiced it does appear to worship the randomized clinical trial (RCT) above all else. In fact, that “plausibility bias” that most physicians have often manifests itself as difficulty believing that there is a problem with EBM, that EBM can go so off the rails when it comes to CAM because it really has no mechanism to take plausibility into account. Indeed, it’s been speculated right here on this very blog that the reason prior plausibility is not built right into EBM, so to speak, is because the founders of EBM suffered from it. They assumed that treatments would not reach the stage of large RCTs if they had not proven themselves plausible first through preclinical evidence in laboratory studies, animal experiments, and studies of pathology and lab tests. Under this view, it simply never occurred to the gods of EBM that something as ridiculous as homeopathy could reach the stage of RCTs because they suffered from plausibility bias that blinded them to the very possibility of that happening!
Whether that’s true or not, I don’t know, but it would explain a lot. Either way, as we have pointed out, SBM tries to restore to EBM what it is missing: A consideration of prior plausibility based on scientific considerations. In practice, this is more useful for eliminating incredibly implausible treatments, such as homeopathy and reiki, than it is for putting hard numbers on prior plausibility for treatments because it is not always necessary to estimate a pre-trial probability of success, except when it so low that it would take an incredible amount of evidence to overturn existing knowledge, as it would for homeopathy or reiki. Here’s my plausibility bias: For something like homeopathy or reiki, either of which would require the rewriting of huge swaths of science to become plausible, I consider it reasonable to require supporting evidence at least in the same order of magnitude of quantity and quality as the evidence showing that homeopathy or reiki cannot work to make it reasonable to start to think that either could work. Or, to put it much more simply, extraordinary claims require extraordinary evidence.
That’s my plausibility bias. I’m biased in favor of science and reason and against magical thinking like homeopathy and reiki. You should be biased too.
The homeopaths attack
After I had stopped laughing in response to seeing homeopaths lecture scientists on what is and is not scientific, I delved into the paper. Rutten et al try (and fail—after all they are homeopaths) to establish their scientific bona fides righ in the second paragraph:
The authors of the present paper are doctors and scientists with an interest in homeopathy, committed to the scientific method in researching and practising it. We are qualified in medicine and science and started practising these in conventional contexts, gradually becoming convinced that homeopathy is an effective option, supplementary to rather than conflicting with conventional medicine. We concur with Hansen and Kappel that the disagreement concerning the interpretation of reviews of randomised controlled trials (RCTs) is rooted in prior beliefs and their influence on the perception of evidence. We do not concur, however, with their assumption that the homeopathy community’s positive view of the evidence is due to a rejection of the naturalistic scientific outlook. We ourselves, for example, do not reject any part of the naturalistic outlook.
My first temptation was to point out that the very fact that they are homeopaths means that they are either deluding themselves or lying when they claim that they do not reject any part of the naturalistic outlook. Homeopathy, after all, is rooted in the principles of sympathetic magic, not science. For instance, homeopathy’s law of similars (“like cures like”) is uncannily similar to Sir James George Frazer’s Law of Similarity as described in The Golden Bough (1922) as one of the implicit principles of magic. In addition, the concept that water can somehow retain the imprint of substances with which it’s been in contact, which really underlies the belief among homeopaths that remedies diluted to nonexistence (basically anything diluted more than around 12 C—14C or 15C, to be safe) can have biological effects, is very much like the Law of Contagion. Read the following passage from The Golden Bough and tell me that it doesn’t sound almost exactly like homeopathy:
If we analyse the principles of thought on which magic is based, they will probably be found to resolve themselves into two: first, that like produces like, or that an effect resembles its cause; and, second, that things which have once been in contact with each other continue to act on each other at a distance after the physical contact has been severed. The former principle may be called the Law of Similarity, the latter the Law of Contact or Contagion. From the first of these principles, namely the Law of Similarity, the magician infers that he can produce any effect he desires merely by imitating it: from the second he infers that whatever he does to a material object will affect equally the person with whom the object was once in contact, whether it formed part of his body or not. Charms based on the Law of Similarity may be called Homoeopathic or Imitative Magic. Charms based on the Law of Contact or Contagion may be called Contagious Magic.
A later passage by Sir Frazer is an excellent criticism of the two pillars of homeopathy:
Homoeopathic magic is founded on the association of ideas by similarity: contagious magic is founded on the association of ideas by contiguity. Homoeopathic magic commits the mistake of assuming that things which resemble each other are the same: contagious magic commits the mistake of assuming that things which have once been in contact with each other are always in contact. But in practice the two branches are often combined; or, to be more exact, while homoeopathic or imitative magic may be practised by itself, contagious magic will generally be found to involve an application of the homoeopathic or imitative principle.
See what I mean when I say that the ideas behind homeopathy resemble sympathetic magic far more than they resemble science? From my perspective, all homeopaths—and I do mean all homeopaths—hold views that reject science, no matter how much they fool themselves into thinking they are scientific and buy into the naturalistic world view. I could go on to demonstrate how much of homeopathy is rooted in prescientific vitalism, using Samuel Hahnemann’s own words, but I think you get the idea. Homeopathy is magic water made magic using thought processes akin to those used in voodoo when voodoo practitioners make voodoo dolls.
It is also rather interesting how Rutten et al are so willing to accept science when it comes to RCT evidence but reject the much larger and far more robust body of science that underlies the pre-trial assessment of prior probability that says that homeopathy can’t work. They willfully reject the concept that extraordinary claims require extraordinary evidence, and homeopathy is nothing if not a highly extraordinary set of claims. Instead, Rutten et al make an analogy to crossword puzzles. This analogy is actually rather apt, but not in the way our unhappy homeopaths think it is. Basically, here is the analogy as described by Rutten et al:
Sometimes new evidence overturns theory, but sometimes not; the context is crucial. This has been expressed in terms of a crossword analogy (Haack 1998): the correctness of an entry in a crossword depends upon how well it is supported by the clue, whether it fits with intersecting entries, how reasonable those other entries are, and how much of the crossword has been completed. In this analogy, for homeopathy, the primary entry is: “Does it work (other than by placebo effects)?” The secondary intersecting entries are concerned with “How does, or could, it work?”
Although Rutten et al will never admit it, this analogy is an excellent one for why the occasional “positive” clinical trial of homeopathy does not overthrow the existing scientific paradigm that concludes that homeopathy can’t work, that it is nothing but water, and that any apparently positive effects seen are due either to placebo, random chance, or bias and/or shortcomings in the RCTs. Such trials do not fit with “multiple intersecting entries” in physics, chemistry, and biology that are all consistent with the impossibility of homeopathy; i.e., they do not fit into the crossword puzzle. The only way they could be made to fit into the crossword puzzle would be if homeopathy were shown in a reproducible fashion to cure incurable diseases, such as metastatic pancreatic cancer, in which case homeopathy might go into the crossword puzzle and force the puzzle solver to start rethinking other answers to fit with homeopathy.
In other words, clinical evidence could make us question the rest of the “crossword puzzle” but only if it’s clinical evidence that is so extraordinary in result, quality, and quantity that it starts to rival the existing evidence from multiple disciplines that do not support homeopathy. No such evidence exists for homeopathy, and, in fact, the overall weight of the clinical evidence is consistent with homeopathy not working any more effectively than placebo. Indeed, Ruten et al wrongly relegate the question of how homeopathy could work to a secondary question, and here’s why: When, for a therapy to work the very laws of physics would have to be, as I say so often, not just wrong but spectacularly wrong, the question of how it could work is not secondary. This is in marked contrast to drugs (which inevitably work by either binding to a biological molecule or otherwise reacting somehow), in which case not knowing the exact mechanism is not as concerning. Even cases like the discovery that H. pylori causes duodenal ulcers is not a refutation of this principle with respect to homeopathy. After all, as implausible as the hypothesis that it was a particular bacterial species that was responsible for peptic ulcers in many cases, it did not require the violation of the laws of physics to imagine that a bacterial infection could somehow cause ulcers.
Rutten et al spend considerable verbiage listing the usual suspects for homeopathy, including old meta-analyses, various clinical trials, and, of course the infamous basophil degranulation experiments by Jacques Benveniste. These have been fodder many times before on this blog; so I don’t really want to dwell on them other than to note that in particular Rutten et al reserve most of their vitriol for a meta-analysis and systematic review of the literature by Shang et al published several years ago in The Lancet that found that homeopathy effects are placebo effects. Basically, Rutten et al basically rehash Rutten’s criticisms of Shang’s analysis. These are criticisms I dealt with in detail, and four years of aging don’t make them any better. In fact, the apologia based on “clinical evidence” is nothing that we haven’t heard before and nothing worth rehashing here (other than a link to my previous deconstruction) because the point of Rutten et al is to attack what they call “plausibility bias.” All the trotting out of clinical evidence that allegedly supports homeopathy is in reality a massively flawed lead-in, a thin mint wafer to cleanse the palate, so to speak, to the main argument, which is based on how Shang’s meta-analysis and other clinical trials allegedly support homeopathy but are often cited as evidence against homeopathy.
First, Rutten et al distinguish between homeopathic dilutions in which there might still be some of the original remedy left (generally less than 12C or so, but in reality any homeopathic dilution that gets higher than 7C (10-14) is probably in the femtomolar range or lower, and there aren’t very many substances that have significant biological effects at such a low concentration. None of this stops Rutten et al from proclaiming:
There are obvious sources of pre-trial belief. These include well documented paradoxical low-dilution effects. The basic idea of homeopathy is the exploitation of the paradoxical secondary effects of low doses of drugs. Secondly, reverse or paradoxical effects of drugs and toxins in living organisms as a function of dose or time are very widely observed in pharmacology and toxicology. They are variously referred to as hormesis (the stimulatory or beneficial effects of small doses of toxins) hormligosis, Arndt- Schulz effects, rebound effects, dose-dependent reverse effects and paradoxical pharmacology (Calabrese and Blain 2005; Calabrese et al. 2006; Bond 2001; Teixeira 2007, 2011).
Repeat after me: Hormesis does not justify homeopathy. It’s an analogy that homeopaths love because it’s a hypothesis that states that some substances that are toxic at high doses might be benign or even beneficial at lower doses. (Look back to the fun I had with Ann Coulter’s invocation of hormesis to try to convince you that radiation from the Fukushima nuclear reactor is in fact good for you for an explanation.) This is, of course, wishful thinking on the part of homeopaths, representing extreme over-extrapolation. Hormesis might apply to low doses, but much of homeopathy involves no dose; i.e., dilution far, far beyond the point where it is highly unlikely that even a single molecule of the original substance remains. Rutten et al try to dodge this question by claiming that most homeopathic remedies are not “ultramolecular dilutions” (i.e., dilutions far beyond Avogadro’s number that leave nothing behind). Even if that’s true, many homeopathic dilutions are “ultramolecular” dilutions, and homeopathy does postulate that dilution and succussion do increase the potency of homeopathic remedies. Have Rutten et al forgotten the Law of Infinitesimals?
They haven’t, though. After trying to argue that most homeopathic remedies are not “ultramolecular,” Rutten et al then cite a bunch of dubious in vitro studies claiming that ultramolecular dilutions can have biological effects. I’ve looked at many such studies (for instance, this study of homeopathic remedies on human breast cancer cell lines), and quite often what you find is shoddy methodology, effects of solvents and contaminants, and other potential explanations for the observed results that do not involve having to throw out huge swaths of physics and chemistry. Amusingly, Rutten et al even admit that such results have a serious problem:
A more recent meta-analysis evaluated 67 in vitro biological experiments in 75 research publications and found high-potency effects were reported in nearly 75 % of all replicated studies; however, no positive result was stable enough to be reproduced by all investigators (Witt et al. 2007).
Can you say “publication bias”? Sure, I knew you could.
Can you also say: Anecdotal evidence? Sure, I knew you could:
The other major source of our prior beliefs is practice experience. This may be regarded the lowest level of evidence, but it is under-rated by many (Vandenbroucke 2001). After adding homeopathy to conventional treatment, many unsuccessful cases improved (Marian et al. 2008). The repetitive character of such experiences gradually updated our belief, consistent with Bayesian theory (Rutten 2008).
In other words, Rutten et al admitting that the source of their “positive plausibility bias” towards homeopathy is based on anecdotes. That is, after all, what “practice experience” is: Anecdotes, confirmation bias, and the like. It’s the same reason that Dr. Jay Gordon, for instance, believes that vaccines cause autism when the evidence from large epidemiological studies does not support that belief. He sees what he thinks are cases of “vaccine injury” manifesting itself as autism and, because he believes that vaccines cause autism, attributes his patients’ autism to vaccines. Rutten et al also cite non-blinded, non-randomized “real world” (pragmatic) trials as contributing to their pre-test plausibility bias towards homeopathy.
Pre-trial belief: Science versus anecdote
We have argued that EBM has a shortcoming, and that shortcoming is that EBM does not adequately consider prior probability in assessing evidence. In EBM, clinical evidence is all, and evidence from RCTs (or even better, meta-analyses or systematic reviews of RCTs) rules the heap. This is not unreasonable when RCTs are only performed for hypotheses that have been developed through a scientific process that takes preclinical observations and builds upon them, such that existing evidence deems them reasonably plausible. CAM in general and homeopathy in particular are not such a case. RCTs of homeopathy in essence measure noise, but only positive noise. Some studies will appear to be positive, and publication bias will make sure that the studies where patients receiving homeopathy do worse are unlikely to be published so that we see in the literature only negative studies or studies apparently positive due either to random chance, either alone or combined with poor study design and/or bias. We and others have proposed taking prior probability into consideration, both for deciding what hypotheses to test in clinical trials and how to interpret the results of existing clinical trials.
The fact is that we have always taken plausibility into account in deciding which clinical trials to perform. We have to because we don’t have unlimited resources, human subjects, or researchers to test in an RCT every hypothesis that comes along. We just don’t. In fact, our resources are currently more constrained than they have been in at least 20 years, with NIH pay lines hovering around the 7th percentile in some institutes. Moreover, the very foundations of medical ethics as laid down in the Helsinki declaration require that human subjects experimentation have a strong background of basic science backing it up. The question is: How do we want to prioritize which trials get done? On what do we base our estimates of prior plausibility that color our decisions regarding which clinical trials to carry out and how to interpret data from existing clinical trials? Homeopaths like Rutten and colleagues would propose that we base our estimate of prior plausibility on anecdote, magical thinking, and dubious in vitro and clinical trial evidence, ignoring the massive, well-established prior implausibility of homeopathy that a rational scientific assessment will arrive at. Scientists base their assessment of prior plausibility based on as objective as possible an interpretation of existing scientific data.
I know which one I would choose.
I also have a message for Rutten and is merry band of homeopaths. You accuse us of “plausibility bias” as though that were a bad thing. It’s not. As Mark Crislip pointed out, what plausibility bias should really be called is reality bias. We are biased towards reality. Homeopaths are biased towards what they think is reality but is in actuality magical thinking.
Again, I know which one I choose.
Finally, we don’t have unlimited resources to test every hypothesis that anyone can think up. There isn’t the money. There aren’t enough scientists. Even leaving aside the serious ethical problems that come with testing highly improbable remedies on human subjects, there aren’t enough human subjects to test the promising drugs that have a reasonable probability of working (i.e., of being efficacious and safe) based on preclinical testing. Resource constraints have always existed, and scientists have never just tested whatever the heck they felt like testing. Plausibility has always been a major part of deciding which experiments to do, which promising compounds to take to clinical trials, which treatments to try. Think of it this way: We could estimate plausibility as carefully as we can based on scientific testing, evidence published in the existing scientific literature, and data from small pilot clinical trials. Or, taking the approach of Rutten et al, we can estimate plausibility from anecdotal experience, questionable experiments and clinical trials, and considerations that completely ignore the laws of physics and chemistry.
Again, I know which method I choose.

   Homeopathy and Plausibility
Steven Novella 
Link
The fundamental concept of science-based medicine (SBM) is that medical practice should be based upon the best available science. This may seem obvious, but there are many important details to its application, such as the relationship between clinical and basic science. Clinical claims require clinical evidence, but clinical evidence can be tricky and is often preliminary. It is therefore helpful (I would say essential) to view the clinical evidence in light of all of the rest of science.
A thorough basic and clinical science analysis of a medical claim can be summarized by the term “plausibility,” or “prior probability” if you want to put it into statistical terms. When we say a certain belief is plausible we mean it is consistent with what we know from the rest of science. In other words, because of the many weaknesses of clinical evidence, in order for a therapy to be generally accepted as part of SBM it should have a certain minimal supporting clinical evidence and overall scientific plausibility.
These can exist in different proportions – for example one therapy may be highly plausible (it would be shocking if it were not true) and have modest supporting clinical evidence, while another may have unknown plausibility but with solid clinical evidence of efficacy. But no therapy should have clinical evidence that suggests lack of efficacy, nor extreme implausibility (not just an unknown mechanism, but no possible mechanism).
The poster child for extreme scientific implausibility within medicine is, arguable, homeopathy. Its “law of similars” is little more than sympathetic magic, and its “law of infinitessimals” leads to concoctions diluted to the point that they have zero active ingredient and only the magical memory left behind. It is this reality which inspired a recent review to characterize homeopathy as “witchcraft.”
What about the clinical evidence? The clinical evidence, when taken as a whole, and even if we set aside the question of plausibility, shows that homeopathy does not work. Decades of research have failed to provide supporting evidence for any application of homeopathy. It has failed to reject the null hypothesis, to show efficacy, to allow for the recommendation of homeopathy for any indication, to differentiate homeopathy from a placebo – in other words, homeopathy does not work.
If we consider the SBM criteria above, we can summarize homeopathy by saying it has extreme implausibility and the clinical evidence shows lack of efficacy. It should not work, and it does not work. There is no legitimate controversy about this. The only think keeping homeopathy alive are delusional proponents and a public (including many regulators) who do not know what homeopathy truly is.
Proponents, however, are desperately trying to keep their pseudoscience alive by misrepresenting the evidence and the arguments of homeopathy critics. A recent example of this is a paper by Peter Fisher, the Clinical Director Royal London Hospital for Integrated Medicine, and a prominent homeopath. He is trying to coin the phrase “negative plausibility bias.” His argument, essentially, is that the evidence shows homeopathy works (at least as well as medical treatments that do not work, which is an odd argument), but there is a negative plausibility bias against homeopathy which motivates scientists to reject this evidence. Fisher is both wrong and irrelevant in this position.
Fisher is wrong in arguing that the clinical evidence supports the efficacy of homeopathy. He does this by cherry picking positive studies (which are part of the noise of any clinical research), a common strategy. Meanwhile, systematic reviews do not show supporting evidence for homeopathy. Worse for homeopathy, there is a clear pattern in the research. The better designed and controlled the study, the more negative the results – a clear pattern of lack of efficacy. Even reviews that desperately try to spin the results in a positive way in the end show there is no evidence to support the efficacy of homeopathy.
The main point of the article, however, is to dismiss the scientific assessment of homeopathy as a “bias.” It seems like it is news to Fisher that plausibility is not a bias – it’s science.
Interestingly, Fisher claims that the law of similars part of homeopathy is not even controversial – and then he makes the same, tired analogies to hormesis and paradoxical drug effects, which actually have no relevance to homeopathy. Homeopathy “ingredients” are chosen for magical reasons that have no relationship to actual biology or chemistry. Further, the high dilutions of homeopathic preparations render such analogies invalid. Fisher and other homeopaths are just fishing for any possible hand-waving justification for homeopathy, but they lend nothing to its plausibility because they fail to make a scientifically coherent argument.
Fisher then essentially argues that the clinical evidence shows homeopathy works (wrong) but is rejected because of the “negative plausibility bias.” There is a kernel of truth to his view in that, in the face of extreme scientific implausibility, even modestly positive clinical evidence is looked upon as insufficient and not definitive. We can look at it this way – what are the odds that a mountain of solid basic science is wrong vs some sloppy and tricky clinical research is wrong? It would take overwhelmingly rigorous and positive clinical evidence to call into question long established principles of basic science. Homeopathy does not come anywhere close – even if we take the distorted and incorrect view of the clinical evidence Fisher is pushing.
The article is essentially Fisher whining that the scientific community is not ignoring the extreme scientific implausibility of homeopathy.
Fisher also tries to make a tu quoque argument – recycling yet another old ploy of medical pseudoscientists. He says that the evidence for antibiotic use in upper respiratory tract infections (URTI) is no different than homeopathy but practitioners use antibiotics and not homeopathy. The evidence base for any other practice is irrelevant to homeopathy, but even that aside his argument is a curious one. I agree that systematic reviews fail to show efficacy for routine antibiotic use in URTI. Therefore – they should not be used. In fact there are efforts within mainstream medicine to reduce the use of antibiotics in URTI, and to eliminate their routine use.
The story with URTI is more complex, however, because some people do get bacterial interference with URTI and there may be a role for antibiotics in selected cases – the trick is in knowing how to select those cases. More research is legitimately needed to explore these issues.
The only consistent position, therefore, is to favor the elimination of routine antibiotic use in URTI, based upon current evidence, and also to favor the complete elimination of homeopathy as a practice. Meanwhile, it is reasonable to do more research into a possible limited role of antibiotics in selected cases (based partly on plausibility). It is also reasonable to favor the abandonment of any further research into homeopathy, based on its extreme scientific implausibility.
You can call it a “negative plausibility bias” or you can call it science, based upon your perspective.

Evidence Thresholds
Steven Novella on March 27, 2013
Link
Defenders of science-based medicine are often confronted with the question (challenged, really): what would it take to convince you that “my sacred cow treatment” works? The challenge contains a thinly veiled accusation — no amount of evidence would convince you because you are a nasty skeptic.
There is a threshold of evidence that would convince me of just about anything, however. In fact, I have been convinced that many scientific claims are likely to be true — sufficiently convinced to act upon the conclusion that they are true. In medicine this means that I am convinced enough to use them as a basis for medical practice.
There are many functional differences between practitioners of SBM and those who accept claims and practices that we would consider to be pseudoscience or fraud, but I was recently struck by one particular such difference — where we set the threshold of evidence before accepting a claim.
Last week I took part in a debate about the legitimacy of homeopathy (you can read my full account here and here). On the other side was Andre Saine, a Canadian naturopath and Dean of the Canadian Academy of Homeopathy. If I had to summarize the key difference between Saine’s position and my own during the debate it was that he accepted incredibly poor evidence as sufficient to establish the reality of homeopathy. His threshold of evidence was astoundingly low.
At the same time he expressed disbelief that skeptics could maintain their skepticism in light of the evidence he presented. I could come to no other conclusion than that Saine simply has no concept of the usual threshold of evidence for acceptance in medicine and in mainstream science generally.
What Would It Take?
What convincing evidence looks like is something we have thought about and written about extensively. I also write about scientific topics outside of medicine, and this has helped give me a broader perspective on this question as well. Proponents of ESP, for example, also accept a lower standard of evidence.
The question is – at what point should the scientific community generally accept the reality of a phenomenon? This means that the alternate explanation, that the positive data is flawed or misleading in some way, can be confidently ruled out.
Here are the four criteria that need to be simultaneously met in order to be considered compelling scientific evidence:
1- Methodologically rigorous, properly blinded, and sufficiently powered studies that adequately define and control for all relevant variables (confirmed by surviving peer-review and post-publication analysis).
2- Positive results that are statistically significant.
3- A reasonable signal to noise ratio (clinically significant for medical studies, or generally well within our ability to confidently detect).
4- Independently reproducible. No matter who repeats the experiment, the effect is reliably detected.
What we often see with dubious medicine (like homeopathy) is that only criteria 2 is necessary – any study that shows statistical significance is taken as iron-clad.
We also often see a shell game similar to buying a new car. Car salespeople often use the four square method – they divide a sheet of paper into four squares – in one is the price of the car, in the other is the loan rate, in the third is the down payment, and in the fourth is the money they will give you on the trade-in. These will all calculate to your monthly payment.
But here’s the trick – the car dealer will use this method to make sure they make their profit. If they give you a good deal on the trade-in, then they don’t give you a good deal on the price of the car. You can never get a good deal on all four squares at the same time.
Proponents of dubious science work the same way – they offer studies that have one or maybe two of the above criteria, but never all four at the same time. They may offer a poorly designed study with positive results, or a well designed study with positive results, but clinically insignificant and unable to be replicated.
There is a reason why you cannot get all four criteria at the same time – because the phenomenon in question is not real. Only a real effect would show up consistently in highly rigorous studies.
It also needs to be pointed out that meeting these criteria is the baseline for scientific acceptance, without even considering prior plausibility. Within each criterion there is a range of quality – how rigorous a study, replicated how many times, with what effect size? The more implausible a claim, the greater the threshold should be in order to overcome that implausibility.
Homeopaths and proponents of implausible claims don’t like this reasoning. They deride it as “plausibility bias.” Everyone else calls it “science.”
But it is important to point out that even without considering prior probability or plausibility, homeopathy still fails to meet even the minimal scientific criteria for acceptance. It is not even close – even if we give it every benefit of the doubt.
Defending the Threshold
If you are convinced by the reality of something like homeopathy, acupuncture, energy medicine, colonic therapy, or something equally unlikely, the threshold of acceptance seems unfair and broken. It seems like a trick nasty skeptics use to deny the reality of your fabulous medicine.
It is, rather, the standard threshold of acceptance within mainstream science (obviously there is a range within that standard, but it is at least a minimal threshold).
Part of the philosophy of science-based medicine is that such a rigorous standard is justified and necessary, and in fact it needs to be even higher than perhaps is currently practiced in medicine. We have discussed many reasons why this is the case, each one can be the topic of a lengthy article, but I will just summarize them here.
– Medical research is challenging as people are generally a variable and noisy system in which to conduct studies and control variables.
– Placebo effects are multifarious and difficult to account for completely.
– Researcher degrees of freedom make it possible to manufacture positive results even out of a completely non-existent phenomenon. This requires special rigor in study design and execution to avoid, and also independent replication.
– Most published studies are wrong, because most new ideas in medicine do not pan out, and most studies are preliminary and are therefore subject to significant researcher bias in the positive direction.
– There is occasional fraud in scientific research.
– Publication bias distorts the overall scientific literature.
– There is considerable financial bias in medical research, as this is an applied science and often billions of dollars ride on the results of research.
– Humans generally are subject to a host of cognitive biases, heuristics, logical fallacies, errors in memory and perceptions, and other mechanisms of self-deception. We can naively be led to believe almost anything with utmost confidence.
Conclusion
Rigorous science anchors us to reality. Without it our beliefs will drift off into a fantasy world that caters to our emotions and desires but has little connection to reality. We might end up believing that pure water can retain the “essence” of a substance that was once diluted in it, and that this essence can heal people based upon completely unrelated features, such as their personality type.
People, left to their own devices, will believe in magic. Such a tendency is our evolutionary inheritance. But so is the capacity for logic and critical thinking.
Over the past two centuries, as scientific medicine has matured, we have learned to apply greater and greater rigor to the study of medicine and disease. We have learned more about our capacity for self-deception, and the subtle ways to manipulate data and research.
We now know what it takes to prove that something is really real, and doesn’t just seem real. We should vigorously resist those who wish to discard this hard-won wisdom because it threatens their cherished belief.

Beware The P-Value
The p-value was meant to be used as a convenient and quick test to evaluate how likely a result was due to chance, or a real effect. It has since come to be treated as an indication of importance or truth, particularly in the CAM world. This is a problem.
Steven Novella on July 2, 2014
Tweet


Part of the mission of SBM is to continually prod discussion and examination of the relationship between science and medicine, with special attention on those beliefs and movements within medicine that we feel run counter to science and good medical practice. Chief among them is so-called complementary and alternative medicine (CAM) – although proponents are constantly tweaking the branding, for convenience I will simply refer to it as CAM.
Within academia I have found that CAM is promoted largely below the radar, with the deliberate absence of public debate and discussion. I have been told this directly, and that the reason is to avoid controversy. This stance assumes that CAM is a good thing and that any controversy would be unjustified, perhaps the result of bigotry rather than reason. It’s sad to see how successful this campaign has been, even among my fellow academics and scientists who should know better.
The reality is that CAM is fatally flawed in both philosophy and practice, and the claims of CAM proponents wither under direct light. I take some small solace in the observation that CAM is starting to be the victim of its own success – growing awareness of CAM is shedding some inevitable light on what it actually is. Further, because CAM proponents are constantly trying to bend and even break the rules of science, this forces a close examination of what those rules should actually be, how they work, and their strengths and weaknesses.
This brings me to the specific topic of this article – the dreaded p-value. The p-value is a frequentist statistical measure of the data of a study. Unfortunately it has come to be looked at (by non-statisticians) as the one measure of whether or not the phenomenon being studied is likely to be real, even though that is not what it is and is never what it was meant to be.
As an aside, this trend was likely driven by the need for simplicity. People want there to be one simple bottom line to a study, so they treat the p-value that way. It’s like evaluating the power of a computer system solely by its clock speed, one number, rather than considering all the components.
A recent paper by Pandolfi and Carreras nicely deconstructs the myth of the p-value and show how p-value abuse is especially problematic within the world of CAM (thanks to Mark Crislip for bringing this paper to my attention) – “The faulty statistics of complementary alternative medicine (CAM)“.
For background, the p-value is the probability that the data in an experiment would demonstrate as much or more of a difference between the intervention and control give the null hypothesis. In clinical studies we can rephrase this to: what is the probability that the treatment group would be as different from the control group or more assuming the treatment has no actual effect? Many people, however, misinterpret the p-value as meaning – what is the probability that the treatment works.
Pandolfi and Carreras correctly point out that this is committing a formal logical fallacy, the fallacy of the transposed conditional. To illustrate this they give an excellent example. The probability of having red spots in a patient with measles is not the same as the probability of measles in someone who has red spots.
In other words, the p-value tells us the probability of the data given the null hypothesis, but what we really want to know is the probability of the hypothesis given the data. We can’t reverse the logic of p-values simply because we want to.
No worries – Bayes Theorem comes to the rescue. This is precisely why we at SBM have largely advocated taking a Bayesian approach to scientific questions. A Bayesian approach is ironically how people generally operate. We have prior beliefs about the world, and we update those beliefs as new information comes in (unless, of course, we have an emotional attachment to those prior beliefs, but that’s another article).
In science the logic of the Bayesian analysis is essentially: establish a prior probability for the hypothesis, then look at the new data and calculate how that new data affects the prior probability, giving a post probability.
Pandolfi and Carreras point out that, ironically, this is how doctors function in everyday clinical thinking. When we see a patient we determine the differential diagnosis, a list of possible diagnoses from most likely to least likely. When we order a diagnostic test for a specific diagnosis on the list, we first consider the pre-test probability of the diagnosis. This is based upon the prevalence of the disease and how closely the patient matches the demographics, signs, and symptoms of that disease. We then apply a diagnostic test that has a certain specificity and sensitivity, and based on the results we determine the posterior probability of the diagnosis.
Therefore, the pre-test probability is essential to determining the likelihood that a diagnostic test is either a false positive vs a true positive, or a false negative vs a true negative. You can’t properly interpret the results of the test without knowing the pre-test probability.
Ironically, this logic is abandoned when evaluating scientific research. In fact, the main flaw in the way evidence-based medicine is applied is that it ignores the pre-test probability, and relies heavily on an indirect measure (the p-value) in isolation to interpret test results. If applied to clinical medicine, such a process would constitute gross malpractice.
To drive this point home a little further, using a p-value in isolation in a clinical study to determine if the phenomenon under study is real is like using a non-specific diagnostic test to determine that a patient has a very rare disease, ignoring predictive value and the possibility of a false positive test. As experienced clinicians understand, if a disease is truly rare, then even a reasonably specific test is far more likely to generate a false positive than a true positive.
The analogy here is this – when studying a phenomenon that is unlikely, a significant p-value is far more likely to be a false positive than a true positive. This is why p-values are especially problematic when applied to CAM.
CAM modalities are alternative largely because they did not emerge from mainstream scientific thinking. In many cases, the claims made are incompatible with modern science. Homeopathy, for example, would require rewriting the physics, chemistry, physiology, and biology textbooks to a significant degree. Apparently violating basic laws of science at the very least renders a hypothesis equivalent to a rare disease – having a low prior probability. Therefore, even with an impressive looking p-value of 0.01, the probability could still be overwhelming that the phenomenon being tested is not real and the outcome is a false positive.
Conclusion
The Pandolfi and Carreras paper nicely illustrates one of the core principles of science-based medicine – putting the science back into medicine. Evidence is not enough, we also have to put that evidence into the context of our basic scientific understanding of the world, expressed as a prior probability. It may not be possible to have a rigorous quantitative expression of that prior probability, but we can at least use representative figures.
For example, Pandolfi and Carreras use a prior odds of 9:1 against to represent the skeptical position. This is being generous, in my opinion, as I would give odds of 999:1 at least for claims such as homeopathy. But even using the highly conservative odds of 9:1, even a p-value of 0.01 does not favor the phenomenon being real, but rather the null hypothesis.
The two take-home messages here are these: Don’t rely on p-values as the sole measure of a study’s outcome. Favor, rather, a Bayesian analysis. Even in the absence of a formal Bayesian analysis, an informal Bayesian approach will help put the study results into context.
Second – we should probably raise the bar for statistical significant. A p-value of 0.05 is not as impressive as most people might think. Mark suggested we set the bar at 0.001 as a first approximation, and then we go down from there based upon prior probability.
Doing this will work massively against the interests of CAM, because of their low prior probability. But this is in the interests of good medicine.

P Value Under Fire
Steven Novella on March 9, 2016
Link

The greatest strength of science is that it is self-critical. Scientists are not only critical of specific claims and the evidence for those claims, but they are critical of the process of science itself. That criticism is constructive – it is designed to make the process better, more efficient, and more reliable.
One aspect of the process of science that has received intense criticism in the last few years is an over-reliance on P-values, a specific statistical method for analyzing data. This may seem like a wonky technical point, but it actually cuts to the heart of science-based medicine. In a way the P-value is the focal point of much of what we advocate for at SBM.
Recently the American Statistical Association (ASA) put out a position paper in which they specifically warn against misuse of the P-value. This is the first time in their 177 years of existence they have felt the need to put out such a position paper. The reason for this unprecedented act was their feeling that abuse of the P-value is taking the practice of science off course, and a much needed course correction is overdue.
What is a P-value?
A P-value is a specific statistical analysis of data which addresses the following question: what is the probability that the data would be at least as extreme as it is given the null hypothesis (the assumption that there is no real difference between the groups or phenomena being compared)? One of the problems with the P-value is that few scientists and practitioners who use science can give a thoroughly accurate description of what it actually is.
The P-value was developed by Ronald Fisher in the 1930s as a shortcut to determine when data should be taken seriously. It was intended as an understandable statistical method, and unfortunately it worked too well. People, even scientists, crave simplicity, and so the P-value came to be used far beyond its intentions, and is now erroneously and simplistically seen as the one measure of whether or not a hypothesis is likely to be true.
It is like thinking that megapixels is the only thing you need to know about a digital camera in order to understand its quality, or that processor speed is the only number you need to know about a computer.
P-values range from 0 to 1; the lower the number, the lower the probability that the data is a result of chance alone. Traditionally a P-value of 0.05 is the threshold for determining that results are robust enough to be published. This threshold, of course, is entirely arbitrary.
The problem with P-values
The main problem with P-values is that people use them as a substitute for a thorough analysis of the overall scientific rigor of a study. If the P-value creeps over the 0.05 level then people assume the hypothesis is likely to be true – but that is not what the P-value means.
In essence, it is common to turn the P-value on its head, interpreting it as meaning the probability that the hypothesis is true, rather than the probability of the data given the null hypothesis. Let me quote now from the ASA:
Well-reasoned statistical arguments contain much more than the value of a single number and whether that number exceeds an arbitrary threshold. The ASA statement is intended to steer research into a ‘post p<0.05 era.’
Further, an accompanying press release states:
Good statistical practice is an essential component of good scientific practice, the statement observes, and such practice “emphasizes principles of good study design and conduct, a variety of numerical and graphical summaries of data, understanding of the phenomenon under study, interpretation of results in context, complete reporting and proper logical and quantitative understanding of what data summaries mean.”
Those statements are actually a reasonable summary of the principles of science-based medicine.
Let me expand upon the need for “complete reporting.” The ASA is specifically recommending full transparent reporting by scientists of all statistical and data analyses that they performed in doing their study. This is essential, because the P-value assumes that the data being analyzed is the only data created in the study and that it was analyzed in only one way.
Looked at another way, every time a researcher makes another measure, or a different comparison, makes any decision about how to gather and analyze data, or includes data from a prior observation, they are giving themselves another roll of the statistical dice. The P-value of 0.05 becomes entirely meaningless if everything the researchers did is not taken into consideration.
In 2012 Simmons, Nelson, and Simonsohn demonstrated this nicely with what they called, “Researcher degrees of freedom.” They showed how easy it is to manufacture P-values <0.05 just by exploiting common choices researchers make, choices that are often not reported. This phenomenon has been called p-hacking.
P-hacking appears to be pervasive, and is partly to blame for the fact that many published results have a hard time being replicated. Exact replications tend to reduce degrees of freedom, because many of the choices have already been made, and so the p-hacked results vanish.
The ASA offers the following six principles as a fix to the current “P-value crisis”:
1. P-values can indicate how incompatible the data are with a specified statistical model.
2. P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.
3. Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold.
4. Proper inference requires full reporting and transparency.
5. A p-value, or statistical significance, does not measure the size of an effect or the importance of a result.
6. By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis.
To get back to my digital camera analogy, it would be nice to also know how big the CCD is, the size and quality of the lens, whether or not is has a shutter, the capacity and speed of the memory, the features offered in the menu, and the overall quality of the pictures. Megapixels alone do not tell you much.
Likewise, when evaluating a scientific study you should look at the overall rigor of the design and execution: was it properly blinded, was the blinding measured and successful, what was the effect size compared to the variability (signal to noise ratio), was the effect clinically significant and important, what was the dropout rate, are the comparisons fair and reasonable, were the statistics properly done, and what was the power of the study?
But you are also just getting started – how plausible is the hypothesis, what did other similar studies find, have the results ever been independently replicated, and are there any systematic reviews?
For researchers and journal editors this also means requiring that more than just P-values are reported. How about effect sizes, and disclosure of every variable measured and comparison considered. Additional statistical analysis, such as a Bayesian analysis, would also supplement the P-value and put the data into a much more thorough context.
Basically what the ASA and others are recommending is that scientists do what we do here at SBM – thoroughly analyze every aspect of a scientific claim to come up with the best conclusion we can about whether or not a claimed phenomenon is likely to be real. P-values are one tiny aspect of that analysis – a highly overrated aspect.
Some journals have gone so far as to ban the use of P-values in papers they publish. I agree with those who think this is unnecessarily draconian, but I certainly get their point. They want to force the issue, and perhaps they are right that such extreme measures are necessary to wake the scientific community out of their P-value coma.
Conclusion
The ASA’s bold action to put out their first position paper on a fundamental statistical method in their 177 years of existence may signal, finally, the end of the P-value era. I’m not holding my breath because deeply ingrained culture changes only slowly and with difficulty. It is an important step, however, and adds to a host of other steps in that direction.
Moving away from over-reliance on P-values is also in line with what we advocate at SBM. In fact, one of our major criticisms of the practice of evidence-based medicine (EBM) is that is relies too heavily on P-values in clinical studies, without (as the ASA advocates) putting the claims in a full scientific context.
This is precisely why we have been strong advocates of Bayesian analysis, which asks the correct question – what is the probability that the hypothesis is true given this new data?
Relying on an EBM approach and P-values has led to absurd outcomes, such as recommending homeopathic treatments (or at least further study) based upon a few clinical studies. Given the extreme scientific implausibility of homeopathic potions, the most likely explanation for any P-values <0.05 is P-hacking, or just publication bias. Only consistently positive studies with rigorous design, clinically significant results, and independent replication should be worthy of attention.
In medicine over-reliance on P-values has arguably led to the adoption of many treatments that in retrospect were worthless or even counterproductive. Where we set the threshold for adopting new treatments is a vital question for the medical profession, and we need to pay very close attention to how such decisions are made.
Putting the P-value in its place is a much needed correction to this process.

0.05 or 0.005? P-value Wars Continue
The p-value is under fire yet again, but this time with some quick-and-dirty solutions (and some long-and-onerous ones too) to the problems created by relying on this quick-and-dirty test.
Steven Novella on August 2, 2017
Link

Over the years Major League Baseball has tweaked the dimensions of the field, specifically the distance and height of the pitcher’s mound and the area of the strike zone. They did this in order to adjust the balance between pitchers and hitters, mostly to shift the balance toward hitters to make games more exciting for the fans.
Scientists are debating similar tweaks to statistical significance, to adjust the balance between false positives and false negatives. As with pitchers and batters, some changes are a zero-sum game – if you lower false positives, you increase false negatives, and vice versa. Where the perfect balance lies is a complicated question and the increasing subject of debate.
A recent paper (available in preprint) by a long list of authors, including some heavy hitters like John P.A. Ioannidis, suggests that the p-value that is typically used for the threshold of statistical significance, be changes in the psychology and biomedical fields from 0.05 to 0.005. They write:
For fields where the threshold for defining statistical significance for new discoveries is P < 0.05, we propose a change to P < 0.005. This simple step would immediately improve the reproducibility of scientific research in many fields. Results that would currently be called “significant” but do not meet the new threshold should instead be called “suggestive.” While statisticians have known the relative weakness of using P≈0.05 as a threshold for discovery and the proposal to lower it to 0.005 is not new, a critical mass of researchers now endorse this change.
The p-value is defined as the probability of the results of an experiment deviating from the null by as much as they did or greater if the null hypothesis is true. If that sounds difficult to parse, don’t feel bad. Many scientists cannot give the correct technical definition. To put it more simply, what are the odds that you would have gotten the results you did (or greater) if your hypothesis is not true? In medicine this usually refers to an effect, such as the difference in pain reduction between a placebo and an experimental treatment. Is that difference statistically significant? A p-value of 0.05, the traditional threshold, means that there is a 5% chance that you would have obtained those results without there being a real effect. A p-value of 0.005 means there is a 0.5% chance – or a change from 1/20 to 1/200.
There are major problems with over-reliance on the p-value. It was never intended to be the one measure of whether or not an effect is real, but unfortunately the human desire for simplicity has pushed it into that role. Also, people tend to flip its meaning – interpreting it as the odds that the effect is real (rather than the data being what it is). This reversal of meaning, however, is not valid for many reasons. A study with a p-value of 0.05 does not mean that there is a 95% chance the effect is real. There could still be only a tiny probability the effect is real, depending on other factors.
Placing so much importance on the p-value has also demonstrably led to what is called p-hacking. There are subtle (and sometimes not-so-subtle) ways in which researchers can bias the outcome of a study to cross over the magical threshold of 0.05, declare their results significant and get them published. This has further led to a problem with reproducing research and nothing short of flooding the literature with a mass of dubious studies.
Essentially the authors of the new editorial are pointing out that the balance between false positives and false negatives has drifted far from an optimal point. Over the years researchers have figured out how to game the p-value. Combined with tremendous pressure to publish positive findings, and the usual biases we all have, this has led to a glut of preliminary findings that are mostly false positives.
What the authors propose would certainly shift the balance away from false positives. It is a straightforward fix, but I have concerns that it may not be optimal, or even enough by itself. I do like their suggestion that we consider 0.005 to be statistically significant, and anything between 0.05 and 0.005 to be “suggestive.” This is closer to the truth, and would probably help shift the way scientists and the public think about p-values. I have already made this mental shift myself. I do not get excited about results with a p-value near 0.05. It just doesn’t mean that much.
The downside, of course, is that this will increase the number of false negatives. Given how overwhelmed the literature is with false positive studies, however, I think this is a good trade-off. Further, the p-value threshold is not the only variable. The authors suggest that you could increase the size of a study by 70% to essentially keep the false negative rate where it is. In this way research is not entirely a zero-sum game. You can decrease both false positives and false negatives by increasing the study size, or the power of the study.
While true, this can be difficult for many researchers, especially those with marginal funding, such as young researchers. For rare diseases or questions for which it is difficult to recruit patients, even despite good funding, it could be hard to get to the needed numbers to achieve p < 0.005. But to this I say – so what? You can still do your small study and if you get marginal p-values you can even still publish. Just don’t call your results “significant.” Call them “suggestive” instead.
There may be unintended consequences to this change, but given the huge problem with false positive studies I say we make the change and see what happens. We can always do further tweaks if necessary.
Also, I don’t want the focus on where to set the p-value to distract from the deeper question of how useful the p-value is at all. Some journals have gone as far as banning p-values entirely, in favor of other methods of statistical analysis. I think this is draconian, but they have the right idea, to put p-values in their place.
For example, effect sizes are extremely important, but often neglected. More important than the p-value is some measure of the signal to noise ratio. What is the effect size compared to what is being measured and the uncertainty of the outcome? Further, Bayesian analysis can be very useful. A Bayesian analysis asks the actual question most researchers think they are asking – what is the probability of my hypothesis given this new data? In a Nature commentary on this issue they suggest that many researchers don’t have the statistical chops to do a Bayesian analysis. Again I say, so what? The answer is to improve the statistical chops of the average researcher.
That is, in fact, the inherent problem here. Many researchers do not understand the limitations of the p-value, or they succumb to the temptation of relying heavily on this one measure because it is the fast track to statistical significance and publication. Many also do not fully understand the nature of p-hacking and how to avoid it. We need more sophistication in the minimal acceptable statistical analysis and methodology of medical research.
All indications are that the balance has shifted unacceptably to the false positive. We probably need a thorough culture change within the medical research community – publishing fewer but more rigorous studies, and clearly indicating which studies are preliminary, and further eliminating publication and citation bias against negative results.
This one shift in the threshold for statistical significance won’t be enough, but I do think it is a move in the right direction.


Rejecting The Null Hypothesis

Steven Novella on October 29, 2025
Link

In various contexts, for practical, philosophical, and logical reasons, there is a default assumption. In the criminal justice system, for example, someone is presumed innocent until proven guilty. Therefore we must act “as if” someone is innocent until the burden of proof is met that they are indeed guilty of the specific crime of which they are accused. In medicine, if someone has signs or symptoms sufficiently suspicious for being a deadly illness, like cancer, the clinician must assume it is cancer until proven otherwise.
In science it is very well-established where the starting assumption and therefore burden of proof lies – for any new hypothesis, it is assumed to be wrong until it has met a sufficient bar of evidence to be taken seriously, and then to be considered possibly true, and eventually probably true. Scientific hypotheses are not considered true until proven false. The burden is even more extreme because proponents are tasked with trying to prove their own hypothesis wrong, not trying to prove it correct. A hypothesis must survive repeated attempts at such disproof to be considered viable. At least, this is good science.
Scientific ideas also have another burden – they must demonstrate themselves to be superior to other alternatives, especially those with a more elegant explanation. You cannot make your theory work by simply introducing lots of new assumptions. Each assumption adds to the burden of proof, it does not subtract from it. If an alternative with fewer new assumptions is available, you must also show evidence that this alternative is not correct before your hypothesis can take the lead. This is Occam’s razor.
In scientific studies, including medical studies, we call this assumption that the claim being tested is wrong the “null hypothesis”. The evidence presented in the study must be sufficient to “reject the null” is order to be considered positive. Even then, a single study does not meet the burden of proof for any new hypothesis, but it means that the one study does support the hypothesis, or at least does not argue for rejecting it.
In medical studies (but not necessarily in other fields) the threshold for rejecting the null is traditionally set at a p-value of 0.05. This means that if the null hypothesis were true, there is a 5% or greater chance of the data in the study being what it is. This is not the same thing as a 95% chance of the hypothesis being true. It just means that the data passes the statistical smell test. You still need to meet the burden of Occam’s razor and look for other explanations of the data, the rigor of the study, the plausibility of the hypothesis, and what other studies show. Further, many experts (myself included) feel that the 0.05 threshold is too low a bar, favoring false positives too much over false negatives. But either way, it’s really OK as long as you look at all the factors I mentioned, and look at the entire literature and not just a single study.
All this brings me to the main point I wish to make – doctors and medical scientists must look at the entire scientific literature relevant to any individual question and ask, is the totality of evidence sufficient to reject the null hypothesis. I like to frame this as the following question: is it possible or probable that we are living in a world where the null hypothesis is true and this data exists? If it is anything but extremely improbable, bordering on impossible, then you have not rejected the null hypothesis. You should further ask: is there a simpler explanation for this data that does not require inventing a new and unproven phenomenon? If so, you must first reject that simpler explanation.
These questions can be extremely clarifying. For example, we can take a topic such as ESP and ask – after 100 years of research, is the totality of the evidence sufficient to reject the null hypothesis? The answer, in my opinion, is clearly no. There is no ESP paradigm that replicates a consistent effect. It is all, ultimately, noise and poor methodology.
In medicine there is also some low-hanging fruit. The most obvious such claim that has failed to reject the null hypothesis is homeopathy. Proponents of homeopathy have failed to prove any of its underlying components – there is no law of similars, substances do not carry any meaningful “essence”, water cannot be programmed with complex chemical information, and dilution does not make a potion for powerful. Further, after more than a century of research, proponents have failed to demonstrate that homeopathic products are effective for any specific indication. Homeopathy is a string of null hypotheses, and they have not rejected a single one.
In fact, beyond a certain point we can not only say that the null hypothesis (homeopathy is not real and doesn’t work) is not rejected, we can say that it has been confirmed. Everything in science is relative – there is no absolute proof, just the best inference to the most likely conclusion. But at some point we can, for all practical purposes, reject a failed idea. In medicine this means – it is unethical to use this idea as a basis for treatment. It is also wasteful, and likely unethical, to conduct further clinical research. At best, you would need to go back to the basic science drawing board and find evidence to reconsider taking the idea seriously again.
There are also regulatory implications. For something like homeopathy, which has been soundly rejected by science, insurance should not cover it, the FDA should not approve it, and medical professions should not teach or endorse it. Naturopaths do, which is part of the reason they are not a legitimate medical profession.
The other medical claim which is in a similar boat to homeopathy is acupuncture. After decades of research and thousands of studies, we have not been able to reject the null hypothesis with acupuncture for any of its basic or clinical claims. In other words, all of the evidence is compatible with a world in which acupuncture does not work.
As with homeopathy, proponents have failed the reject the null when it comes to the basic underlying components of acupuncture. There is no evidence that chi or any sort of life force exists. There is no evidence to reject the null when it comes to the existence of acupuncture points. Even under the assumption that acupuncture points exist (they don’t), there is insufficient evidence to determine where they are, how big they are, and what they do. Further, there is no evidence that inserting acupuncture needles (either at alleged acupuncture points or elsewhere) does anything other than create local trauma and all the downstream effects of that.
Clinically there is insufficient evidence to reject the null in terms of acupuncture being effective in treating any condition. This requires a bit more context. In medicine we have very clearly defined criteria for showing that an intervention has efficacy – it must demonstrate a statistically significant effect beyond placebo. Acupuncture cannot do this. There is no single application where acupuncture shows clear benefit beyond placebo, despite the claims of proponents.
But there is a lot of complexity here when you dig into the details. Simply search for “acupuncture” in the SBM search field and you will find hundreds of articles detailing all the many reasons why clinical acupuncture research does not ever reach the level of rejecting the null hypothesis (here is as good a summary as any). The biggest problem is the failure to show benefit beyond placebo is adequately blinded studies.
There is a risk in prematurely rejecting the null in medicine. There are obvious problems with using treatments that do not work. But in addition, treatments in medicine have a tendency to be institutionalized, which gives them a certain inertia. It can then become difficult to get rid of them, even when the evidence is heading in that direction. Homeopathy and acupuncture are poster-children for this effect as well – they are institutionalized despite never having rejected the null hypothesis.
We see this to a less degree with things like facilitated communication and repressed memory syndrome. These things do not exist scientifically either, and are actively harmful when used clinically, and yet they have their own institutions.
There is evidence that overall we need to slow down a bit in medicine. We are collectively a bit too eager to use new treatments, before the null hypothesis has been rejected. This can be appropriate for compassionate or experimental care in desperate situations, but not for general use. We need to be a bit more skeptical, and we need to set our thresholds for acceptance higher. That, in essence, is what science-based medicine is all about.


   Acupuncture, the P-Value Fallacy, and Honesty
Kimball Atwood
Link
Credibility alert: the following post contains assertions and speculations by yours truly that are subject to, er, different interpretations by those who actually know what the hell they’re talking about when it comes to statistics. With hat in hand, I thank reader BKsea for calling attention to some of them. I have changed some of the wording—competently, I hope—so as not to poison the minds of less wary readers, but my original faux pas are immortalized in BKsea’s comment.
Lies, Damned Lies, and…
A few days ago my colleague, Dr. Harriet Hall, posted an article about acupuncture treatment for chronic prostatitis/chronic pelvic pain syndrome. She discussed a study that had been performed in Malaysia and reported in the American Journal of Medicine. According to the investigators,
After 10 weeks of treatment, acupuncture proved almost twice as likely as sham treatment to improve CP/CPPS symptoms. Participants receiving acupuncture were 2.4-fold more likely to experience long-term benefit than were participants receiving sham acupuncture.
The primary endpoint was to be “a 6-point decrease in NIH-CSPI total score from baseline to week 10.” At week 10, 32 of 44 subjects (73%) in the acupuncture group had experienced such a decrease, compared to 21 of 45 subjects (47%) in the sham acupuncture group. Although the authors didn’t report these statistics per se, a simple “two-proportion Z-test” (Minitab) yields the following:
Sample X N Sample p
1 32 44 0.727273
2 21 45 0.466667
Difference = p (1) – p (2)
Estimate for difference: 0.260606
95% CI for difference: (0.0642303, 0.456982)
Test for difference = 0 (vs not = 0): Z = 2.60 P-Value = 0.009
Fisher’s exact test: P-Value = 0.017
…
Wow! A P-value of 0.009! That’s some serious statistical significance. Even Fisher’s more conservative “exact test” is substantially less than the 0.05 that we’ve come to associate with “rejecting the null hypothesis,” which in this case is that there was no difference in the proportion of subjects who had experienced a 6-point decrease in NIH-CSPI scores at 10 weeks. Surely there is a big difference between getting “real” acupuncture and getting sham acupuncture if you’ve got chronic prostatitis/chronic pelvic pain syndrome, and this study proves it!
Well, maybe there is a big difference and maybe there isn’t, but this study definitely does not prove that there is. Almost two years ago I posted a series about Bayesian inference. The first post discussed two articles by Steven Goodman of Johns Hopkins:
Goodman SN. Toward evidence-based medical statistics. 1: The P value fallacy. Ann Intern Med. 1999;130:995-1004.
Goodman SN. Toward evidence-based medical statistics. 2: The Bayes factor. Ann Intern Med. 1999;130:1005-13.
I won’t repeat everything from that post; rather, I’ll try to amplify the central problem of “frequentist statistics” (the kind that we’re all used to), and of the P-value in particular. Goodman explained that it is logically impossible for frequentist tools to “both control long-term error rates and judge whether conclusions from individual experiments
[are] true.” In the first article he quoted Neyman and Pearson, the creators of the hypothesis test:
…no test based upon a theory of probability can by itself provide any valuable evidence of the truth or falsehood of a hypothesis.
But we may look at the purpose of tests from another viewpoint. Without hoping to know whether each separate hypothesis is true or false, we may search for rules to govern our behaviour with regard to them, in following which we insure that, in the long run of experience, we shall not often be wrong.
Goodman continued:
It is hard to overstate the importance of this passage. In it, Neyman and Pearson outline the price that must be paid to enjoy the purported benefits of objectivity: We must abandon our ability to measure evidence, or judge truth, in an individual experiment. In practice, this meant reporting only whether or not the results were statistically significant and acting in accordance with that verdict.
…the question is whether we can use a single number, a probability, to represent both the strength of the evidence against the null hypothesis and the frequency of false-positive error under the null hypothesis. If so, then Neyman and Pearson must have erred when they said that we could not both control long-term error rates and judge whether conclusions from individual experiments were true. But they were not wrong; it is not logically possible.
The P Value Fallacy
The idea that the P value can play both of these roles is based on a fallacy: that an event can be viewed simultaneously both from a long-run and a short-run perspective. In the long-run perspective, which is error-based and deductive, we group the observed result together with other outcomes that might have occurred in hypothetical repetitions of the experiment. In the “short run” perspective, which is evidential and inductive, we try to evaluate the meaning of the observed result from a single experiment. If we could combine these perspectives, it would mean that inductive ends (drawing scientific conclusions) could be served with purely deductive methods (objective probability calculations).
These views are not reconcilable because a given result (the short run) can legitimately be included in many different long runs…
It is hard to overstate the importance of that passage. When applied to the acupuncture study under consideration, what it means is that the observed difference between the two proportions, 26%, is only one among many “outcomes that might have occurred in hypothetical repetitions of the experiment.” Look at the Minitab line above that reads “95% CI for difference: (0.0642303, 0.456982).” CI stands for Confidence Interval: in the words of BKsea, “in 95% of repetitions, the 95% CI (which would be different each time) would contain the true value.” We don’t know where, in the 95% CI generated by this trial (between 6.4% and 45.7%), the true difference lies; we don’t even know that the true difference lies within that interval at all (if we did, it would be a 100% Confidence Interval)! Put a different way, there is little reason to believe that the “center” of the Confidence Interval generated by this study, 26%, is the true proportion difference. 26% is merely a result that “can legitimately be included in many different long runs…”
Hence, the P-value fallacy. It is that point—26%—that is used to calculate the P-value, with no basis other than its being as good an estimate, in a sense, as any: it was observed here, so it can’t be impossible; when looked at from the point of view of whatever the true proportion difference is, it has a 95% chance of being within two standard deviations of that value, as do all other possible outcomes (which is why we can say with ’95% confidence’ that the true proportion is within two standard deviations of 26%). You can see that the CI is a better way to report the statistic based on the data, because it doesn’t “privilege” any point within it (even if many people don’t know that), but the CI will also steer us away from being wrong only in “the long run of experience.” CIs, of course, will be different for each observed outcome. The P-value should not be used at all.
Now let’s look at a graph from the acupuncture report:
Figure 3. Mean NIH-CPSI total scores of 89 chronic prostatitis/chronic pelvic pain patients treated with 20 sessions of either acupuncture (n = 44) or sham acupuncture (n = 45) therapy over 10 weeks. Error bars represent the SD. To enter into the study, each participant had a minimum NIH-CPSI total score of at least 15 (range 0-43) on both baseline visits (indicated as the average in the baseline value). The primary criterion for response was at least a 6-point decrease from baseline to week 10 (end of therapy). There was no significant difference between the NIH-CPSI total scores in the acupuncture and sham acupuncture groups at baseline, week 5 (early during therapy), or weeks 14, 22, and 34 (post-therapy) evaluations. *Of 44 participants in the acupuncture group, 32 (72.7%) met the primary response criterion, compared with 21 (46.7%) of 45 participants in the sham acupuncture group (Fisher’s exact test P = .02). ‡At week 34, 14 (31.8%) of 44 acupuncture group participants had long-term responses (with no additional treatment) compared with 6 (13.3%) of 45 sham acupuncture group participants (RR 2.39, 95% CI, 1.0-5.6, Fisher’s exact test P = .04).
Hmmm. I dunno about you, but at first glance what I see are two curves that are pretty similar. They differ “significantly” at only two of the six observation times: week 10 and week 34. Why would there be a difference at 10 weeks (when the treatments ended), no difference at weeks 14 and 22, and then suddenly a difference again? Is it plausible that the delayed reappearance of the difference is a treatment effect? The “error bars” don’t even represent what you’re used to seeing: the 95% CI. Here they represent one standard deviation, not two, and thus only about a 68% CI. Not very convincing, eh?
OK, I’m gonna give this report a little benefit of the doubt. The graph shown here is of mean scores for each group at each time (lower scores are better). That is different from the question of how many subjects benefited in each group, because there could have been a few in the sham group who did especially well and a few in the ‘true’ group who did especially poorly. It is bothersome, though, that this is the only graph in the report, and that the raw data were not reported. Do you find it odd that the number of ‘responders’ in each group diminished over time, even as the mean scores continued to improve?
Just for fun, let’s see what we get if we use the Bayes Factor instead of the P-value as a measure of evidence in this trial. Now we’ll go back to the primary endpoint, not the mean scores. Look at Goodman’s second article:
Minimum Bayes factor = e-Z²/2
At 10 weeks, according to our statistics package, Z = 2.60. Thus the Bayes factor is 0.034, which in Bayes reasoning is “moderate to strong” evidence against the null hypothesis. Not bad, but hardly the “P = 0.009″ that we have been raised on and most of us still cling to. The Bayes factor, of course, is used together with the Prior Probability to arrive at a posterior probability. If you look on p. 1008 of Goodman’s second paper, you’ll see that as strong as this evidence appears at first glance, it would take a prior probability estimate of the acupuncture hypothesis being true of close to 50% to result in a posterior probability (of the null being true) of 5%, our good-ol’ P-value benchmark. Some might be willing to give it that much; I’m not.

The p-hackers toolkit
P-hacking is a common and persistent problem in research, with impacts on the scientific literature, reporting, and practice in medicine. But what is it, really?
David Weinberg on April 30, 2020
Link

[Editor’s note: Scott is busy this week, so we bring you instead a post from past contributor Dr. David Weinberg.]
The topic of p-hacking has come up frequently in the recent writings on Science-Based Medicine. These posts are frequently postmortems of flawed studies, pointing out a variety of practices collectively referred to as p-hacking. The term is working its way into popular vocabulary. But what is p-hacking? Why is it bad?
P-hacking is a mis-use of statistical metric known as a p-value. P-values have a very specific meaning, but even experts have difficulty explaining the concept of p-values to laymen. In this article I will try to give an explanation methods and perils of p-hacking for those without a background in statistics.
Background
Producing novel, impactful research is how investigators get published, get funded, get promoted, and make names for themselves in academia. It is also how medical entrepreneurs gather evidence and regulatory approval for their products. For better or worse, one of the factors that makes research more interesting and impactful is attaining findings that are “statistically significant.” Studies with statistically significant results are more likely to be submitted for publication and, if submitted, more likely to be accepted. Statistical significance is considered a proxy for scientific significance or clinical significance. If conducting research for regulatory approval (such as for the FDA), achieving statistically significant results is likely to be essential for success. The metric that is used as a yardstick for statistical significance is know as a “p-value.”
The obsession with p-values and statistical significance has come under scrutiny by some critics. I will not go into all the criticisms of traditional use of p-values and significance testing but, by following the link here, you can learn more. The pressure to create statistically significant research motivates some investigators to gently “massage” the data or analyses to transform a statistically insignificant result into a statistically significant one. The term “p-hacking” has been coined to collectively describe a variety of dubious practices that investigators may employ to achieve statistically significant results when statistical significance was not truly earned. These techniques violate sound statistical principles, increase false positive results, and exaggerate real positive results, a bias known as truth inflation.
P-hacking techniques may be employed naively by well-meaning investigators who believe they are polishing and presenting data in the best light. The motives may be innocent, but the consequence is the proliferation of false and exaggerated conclusions.
To understand p-hacking one needs a basic understanding of the null hypothesis, statistical testing, and p-values.
The null hypothesis
Medical research often looks for differences between variables, or changes in variables over time: do smokers have a greater risk of lung cancer than nonsmokers? Do diabetics treated with Drug A have lower blood sugars than those treated with Drug B? Do children raised in Springfield have higher IQs than children raised in Shelbyville? Under most circumstances the statistical tests exploring these differences start with the default assumption that there is no difference between the variables of interest. This assumption of no difference is known as “the null hypothesis.” The objective of research is then to collect data and run appropriate statistical analysis. If the results demonstrate a difference between the groups and are persuasive enough, the difference between the groups is declared to be “statistically significant.” The null hypothesis is rejected and differences among the variables are accepted.
Statistical testing and p-values
What findings are “persuasive enough” to reject the null hypothesis? This question is the essence of statistical testing. If we set the bar for persuasiveness too low, we risk rejecting the null hypothesis too easily resulting in unjustified conclusions. We might incorrectly conclude that Springfield High students are smarter than Shelbyville students when they are truly the same. This is known as a false positive (or Type 1) statistical error. If we set the bar for persuasiveness exceedingly high, we make it difficult to reject the null hypothesis when real differences exist (a false negative or Type II error). Although significance testing is employed as a safeguard against false positive results, p-values have a very specific meaning, and are widely misunderstood.
P-values are expressed as decimals on a scale from 0 to 1. The p-value is the likelihood that a particular or a more extreme result would be obtained IFthe null hypothesis is true. Let’s pretend we run an experiment and measure a small difference between the IQs of high school seniors in Springfield and Shelbyville. We run an appropriate statistical test and find a p-value of 0.45. This means that IF students in Springfield and Shelbyville have equal IQs, we would expect to find this particular result or a more extreme difference 45% of the time. For most purposes this would be insufficient evidence for rejecting the null hypothesis and we would not be confident declaring that students in Springfield are smarter than those at Shelbyville. Smaller p-values indicate that a given result is less likely to occur if the null hypothesis is true. It is worth mentioning that failing to obtain a p-value low enough to reject the null hypothesis does notallow us to conclude that the null hypothesis is true. In other words, despite a high p-value, Springfield seniors might actually be smarter than their Shelbyville counterparts, but for whatever reason our tests did not confirm this.
There are many reasons that a study might fail to reject the null hypothesis. It could be the null hypothesis is true, but it could also be a poorly designed study, or insufficient sample size, or a very small but real difference, or just bad luck. The p-value does NOT tell you how likely it is that the null hypothesis is really false or that the alternative hypothesis is really true. In most cases, there is insufficient information to make a reliable calculation of those propositions.
Before performing an experiment researchers are obligated to define a threshold p-value. If the results of the experiment yields results that differ sufficiently from the null hypothesis to generate a p-value at or below the preselected threshold, the null hypothesis is “rejected” and the deviation from the null is declared statistically significant.
The designation of the threshold p-value is somewhat arbitrary but depends on the researchers’ tolerance for false positive and false negative results. A more stringent (lower) threshold will decrease the likelihood of false positive results (i.e. finding a difference when there is not one), but will also increase the likelihood of false negative results (i.e. failing to find a difference that actually exists). For medical research the threshold p-value is almost universally P≤.05.
The p≤.05 standard
Selecting a threshold p-value is a based on a variety of philosophical and practical considerations. Philosophically, we want to avoid false positive results – but there is a tradeoff. A very stringent threshold p-value decreases the chances of false positive findings, but effectively raises the hurdle to validate a truly positive finding. In other words it also decreases true positive results and increases false negative results. This can be overcome by designing larger, more powerful studies. Unfortunately, there are practical limitations to the funding and other resources for biomedical research, so larger studies are not always possible or practical. For better or for worse, a threshold p-value of .05 has become the de-factostandard for much of medical research. The consequence of a threshold p-value of .05 is that in situations where the null hypothesis is true, the research will erroneously reject the null hypothesis in 5% (1 in 20) of studies.
A dicey metaphor
Let us explore the implications of a threshold p value of .05. p-values can range from 0 to 1. we can divide this range into 20 increments like this:
1. ≤.052.>.05-.103.>.10-.15,4.>.15-.25.>.2-.256.>.25-.37. > .3-.358.>.35-.49.>.4-.4510.>.45-.511.>.5-.5512.>.55-.613. >.6-.6514. >.65-.715.>.7-.7516.>.75-.817.>.8-.8518.>.85-.919. >.9-.9520. >.95-1.0
We assign each of these increments to one side of a 20-sided die, as depicted below.
If we are comparing 2 groups that are, in fact equal (the null hypothesis is true), utilizing a threshold p-value of .05, every study is like a roll of that 20-sided die. One out of every 20 rolls of the die will land on the ≤.05 side, and we will erroneously reject the null hypothesis and declare the two groups to be different.
P-hacking in action
Let’s say I run a startup company and have a promising vaccine to prevent those afflicted by a zombie bite from being transformed to the walking dead. I design a clinical trial comparing Zombivax vs placebo. The results of this study will result in the success or demise of my company. At the end of the study 43% of the Zombivax group became zombies compared to 68% of the Placebo group. How confident can I be that the difference between the treatments is real? In the case of our study, the null hypothesis is that Zombivax and Placebo are equally effective (or ineffective) in preventing zombïism. If our numbers for Zombivax vs placebo achieve a P≤.05 we can then declare Zombivax superior to placebo.
We analyze our results and achieve a p-value of 0.09. This means that IF Zombivax and placebo are equal (the null hypothesis) and we were able to run our clinical trial over and over again, we could expect, by chance alone, our results (43% for Zombivaz vs 68% for placebo) or a more extreme result 9% of the time. This result does not meet the traditional threshold of P≤.05, so using conventional standards, we would not be able to declare Zombivax more effective than placebo.
As CEO of the company that makes Zombivax, I am very disappointed that the clinical trial did not achieve statistical significance. I instruct my statisticians to go back and review the study design and analysis to see if any details were done “improperly” that might have resulted in the disappointing P=.09 result. They notice that some of the subjects in the Zomivax group missed one of 3 doses of the vaccine. If they omit those subject from the analysis, the Zombivax group does a little better, now achieving a P value of .07! Also, we suspect that some of the placebo-treated may not have really been bitten. Theses subjects were omitted and the data reanalyzed. This changes the p-value to .11, so this analysis is abandoned.
Now the statisticians detect that the vaccine doesn’t seem to work as well in older subjects. If they limit the analysis to subjects 50 or younger, the results look much better, yielding a p-value of .04! As CEO I give my statisticians a bonus and issue a press-release declaring Zombivax a medical breakthrough.
So what is wrong with exploring changes in the data and analyses to optimize the results? Once the data are known, there are many ways things can be adjusted and manipulated that will change the p-value. If one is so motivated it is possible to explore alternatives, accept those that move the results in a desirable direction and reject those that do not. This enables investigators to transform negative or borderline results into positive ones. This is the essence of p-hacking.
Using the 20-sided die as a metaphor, the clinical trial of Zombivax rolled the die. Unfortunately for our company, the die did not land on the ≤.05 side. It landed on the adjacent side for p-values between .05 and .10. What I instructed my statisticians to do is to kick, nudge, and tilt the table until the die rolls over to the desired result. If the die rolls in the wrong direction, they can just reset the die to the original roll and try something else. With enough motivation and creativity, it is likely that they can get the die to fall on the desired side and declare statistical significance.
If Zombivax was truly worthless, our clinical trial and subsequent p-hacking would be an example of a pure false positive result. If Zombivax was slightly effective, our p-hacking would be an example of “truth inflation,” transforming a small, statistically insignificant result into a larger, statistically significant one.
There are many options in the p-hackers toolkit; too many to mention in this article. I will discuss a couple of the more common ones.
Flexible sample size
When doing research it is traditional to pre-specify the sample size (e.g., number of patients, specimens, test runs, etc.) for the study. Under ideal settings this would be done based on existing clues about the behavior of the groups being compared, and through the use of power calculations to ensure that the planned study has a reasonable chance of finding a real positive, if one exists. Often sample sizes are based on more practical considerations, such as the number of subjects available for study, funding, etc. The p-hacker’s way to do it is to enroll a few patients, run analyze the results, enroll a few more and repeat the analysis. This cycle is repeated until a statistically significance result is achieved. Then enrollment is halted. At first glance this seems like a very efficient way to do a study. Only the minimum number of patients needed to achieve statistical significance are needed.
Here’s the problem.
If you want to minimize false positives, you have to roll die and accept the final lie. During the course of the roll, the die will inevitably roll over multiple sides before it ultimately comes to rest. By repeated enrolling and re-analyzing, it is as if we take intermittent snapshots of the die in motion. If they happen to catch the die with the <.05 side face up, the die is stopped mid-roll, and victory is declared. In order to avoid excess false positives you have to set the parameters in advance and accept the outcome of the roll.
Other researcher degrees of freedom
There are many factors that can be tweaked to manipulate study outcomes and p-values. These have been called “researcher degrees of freedom.” An amusing but cautionary paper demonstrated that motivated manipulation of researcher degrees of freedom can dramatically alter research conclusions to such a degree that even absurd conclusions can be “proven” with statistical significance. Researchers make many decisions when they design a study. What kind of patients, what age range, how many, how long they will be followed, what parameters will be measured, at what points in time, etc. If some patients miss exams or doses of medicine, how that be handled during data analysis? What statistical tests will be used, and on, and on. Ideally, these parameters will be defined before the study is begun. Any deviation from the predefined study plan would have to be disclosed and justified when presenting the study results.
Data dredging and HARKing
I can think of no better example of so-called data dredging than this gem from xkcd:
In the Great Jelly Bean study, authors report the shocking result that green jelly beans are linked to acne, complete with a statistically significant p-value. What they failed to disclose in their press release is that they ran analyses on 20 colors of jelly beans and obtained a “significant” p-value once. If one has a large enough database, and runs enough analyses, one is almost certain to stumble on a relationship that is statistically significant. It is just a matter of numbers. Rolling the 20 sided die over and over is bound to produce “statistically significant’ results by chance alone.
There are legitimate ways to test multiple hypotheses, but they require more stringent p-values to declare statistical significance. Had the jelly bean authors disclosed the multitude of analyses they performed, their results would have earned a yawn, not a headline.
This is closely related to the practice known as HARKing (hypothesizing after results are known). In HARKing, investigators look at the data, run multiple analyses until they find something interesting (and probably statistically significant), then pretend that the results they found were what they had been looking for in the first place. If the Jelly Bean Study authors constructed a rationale that green jelly beans were uniquely suspected to cause acne, and reported their results as a confirmation of this hypothesis while conveniently neglecting to report the other 19 analyses, they would be guilty of HARKing.
Conclusion: The significance of insignificant results
The extent to which p-hacking can manufacture false positive results or exaggerate otherwise insignificant results is limited only by the P-hacker’s persistence and imagination. The results of p-hacking are much more consequential than simply padding an investigator’s resume or accelerating an academic promotion. Research resources are limited. There is not enough funding, laboratory space, investigator time, patients to participate in clinical trials, etc to investigate every hypothesis. P-hacked data leads to the misappropriation of resources to follow leads that appear promising, but ultimately cannot be replicated by investigators doing responsible research and appropriate analysis.
Provocative, p-hacked data can be the “shiny thing” that gets undeserved attention from the public, the press, and Wall Street. Of even greater concern, compelling but dubiously-obtained results may be prematurely accepted into clinical practice. And within the CAM world, charlatans may can use sloppy research to promote worthless and irrational treatments.
There is no clear solution to solve the problem of p-hacking. Better education of investigators could reduce some of the more innocent instances. Greater transparency in reporting research results would disclose potential p-hacking. Deviations from planned data gathering and analysis plans should be disclosed and justified. For clinical trials, registries such as clinicaltrials.gov and alltrials.net are intended to provide transparency in the conduct and reporting of clinical trials. Investigators are supposed to “register” their studies in advance, including critical features of study design and an analysis plan. If used as intended, deviations from the registered and reported study details would be evident, and a red-flag for potential p-hacking. For my specialty, colleagues and I compared published studies to clinical trial registries. We found that registries are being underutilized in ways that greatly undermine their intended value. Sadly, my specialty is not unique in this regard.
Reducing or eliminating the reliance of p-values and the arbitrary dichotomy of statistically significant or insignificant results has been proposed by the American Statistical Association. Some journals have gone as far as banning p-values and significance-testing in paper they publish.
Greater understanding of p-hacking among investigators, journals, peer-reviewers, and consumers of scientific literature will promote more responsible research methodologies and analyses.

But Is It Real?
Why we need more science in medicine.
Steven Novella on September 7, 2022
Link
That is the question that must be at the core of any truly scientific endeavor – is the apparent or alleged phenomenon we are dealing with actually real? That may seem obvious, and with some scientific disciplines it is a given. This is not necessarily the case, however, with medical science. Medicine strives to be an applied science, so it asks a lot of proxy questions like “Does it work?”
This may seem like the same question, and I would argue that it should be, but it is extremely easy to get distracted by how we measure and study clinical outcomes so as to miss this point. This is because we first have to operationally define “work”. When I say a treatment works, what does that mean? Unfortunately, to many, it may mean that someone feels subjectively better when they take the treatment, but that doesn’t cut it.
One way to define Science-Based Medicine is by how we define what it means for a science-based treatment to work. It means that when we look critically at the totality of scientific evidence, it is more likely than not (to some reasonable level of confidence) that there is a specific measurable physiological effect from the treatment. We spend a lot of time fleshing out the details of how to do such an analysis. Here is a quick summary:
First, the treatment has to be at least reasonably plausible (a factor that distinguishes SBM from evidence-based medicine – EBM). That does not mean we have to know the precise mechanism of action but it should mean the treatment does not break fundamental laws of physics, or defy a basic understanding of chemistry or physiology, or necessarily introduces a purely magical phenomenon.
We also need high-quality clinical evidence for efficacy. This means that there are experimental trials which isolate the alleged phenomenon as completely as possible from all other confounding factors. Generally in medicine we refer to these confounding factors as placebo effects – an apparent response to the non-specific aspects of being treated, and therefore such trials are called placebo-controlled.
Before we consider a medical phenomenon to be real I would argue that we need such placebo-controlled trials that are sufficiently rigorous in their design and execution that show a statistically significant and clinically meaningful objective outcome (as objective as possible) that sufficiently replicates. Replication must also be independent – not just confined to one lab, set of researchers, or country. Further, the effect persists over time (does not vanish due to the decline effect), and persists when we look at only the highest quality studies. The effect also persists when controlled for publication bias. I would also now include the stipulation that the effect persists when you only look at studies that have been pre-registered, in order to minimize the effects of P-hacking.
Is that too high a bar? Absolutely not – but with some caveats. In medicine it is extremely important to know that a clinical phenomenon is really real, because not only does that form the basis of treating people, it also helps to confirm or refute our understanding of the underlying biology. That understanding is then used to develop and refine further treatments. If we think a phenomenon is real when it isn’t, that can launch an entire branch of research and clinical practice that is disconnected from reality. That represents a profound waste of limited resources, in addition to potential massive harm caused to patients.
The caveat is that the threshold for using a medical intervention is not the same as the threshold for concluding that scientifically a conclusion is solid and we can now use it as a premise for further research and development. Clinical decision making is based mostly on risk vs benefit, and that analysis often takes place with incomplete scientific information. It is reasonable, for example, to use an experimental treatment on a patient with an otherwise terminal illness.
While a risk vs benefit analysis may favor the conclusion that a particular treatment is reasonable to use (for now), that should not be conflated with the conclusion that some specific mechanism of action is a real phenomenon.
Having said that, I also think that the totality of the evidence strongly suggests that the medical profession as a whole needs to recalibrate its assessment of when a treatment is reasonable to use. The primary problem is that there is a disconnect between the actual predictive value of preliminary evidence, and the perceived predictive value. Specifically, apparently positive preliminary evidence that a treatment works has a much lower predictive value than most clinicians believe. This results in a high percentage of “reversals”, where a common practice based on preliminary evidence is stopped because later more definitive evidence is negative.
These are all fixable problems. We can reconsider where to set the traditional P-value before considering a study positive, to shift some of the false positives toward the true negative. We can universally require pre-registering clinical trials. Journals can require statistical analysis other than p-values, focusing more on effect sizes, number needed to treat, and Bayesian analysis. They can also publish more negative studies, and more exact replications. They can require greater rigor, such as internal replications prior to publishing (again, trying to weed out some of the false positives). Preliminary studies can come with explicit warnings about their preliminary nature and their lack of predictive value. Also the standards of EBM systematic reviews should be higher, and align more with the approach of SBM.
Finally I will point out that all of this is not just theoretical. We see in the real world of medicine what happens when the profession fails to adequately ask and answer the question – is the phenomenon actually real? The proliferation of acupuncture is a great example. I won’t go into the details again here, I have written about it extensively in SBM and elsewhere. Suffice to say, the evidence strongly suggests that acupuncture is not real – there is no underlying phenomenon specific to acupuncture. In fact, acupuncture points do not even exist. Yet it has infiltrated almost every medical discipline, and stands as a prime example of what happens without adequate SBM standards. There are plenty of other examples, such as EMDR in mental health practice.
We must also remember that the world of medicine does not only contain honest brokers. There are those who seek to exploit the weaknesses in the system to promote snake oil and take advantage of desperate patients and a gullible public.

Psychology Journal Bans Significance Testing
Steven Novella on February 25, 2015
Link

This is perhaps the first real crack in the wall for the almost-universal use of the null hypothesis significance testing procedure (NHSTP). The journal, Basic and Applied Social Psychology (BASP), has banned the use of NHSTP and related statistical procedures from their journal. They previously had stated that use of these statistical methods was no longer required but can be optional included. Now they have proceeded to a full ban.
The type of analysis being banned is often called a frequentist analysis, and we have been highly critical in the pages of SBM of overreliance on such methods. This is the iconic p-value where <0.05 is generally considered to be statistically significant.
The process of hypothesis testing and rigorous statistical methods for doing so were worked out in the 1920s. Ronald Fisher developed the statistical methods, while Jerzy Neyman and Egon Pearson developed the process of hypothesis testing. They certainly deserve a great deal of credit for their role in crafting modern scientific procedures and making them far more quantitative and rigorous.
However, the p-value was never meant to be the sole measure of whether or not a particular hypothesis is true. Rather it was meant only as a measure of whether or not the data should be taken seriously. Further, the p-value is widely misunderstood. The precise definition is:
The p value is the probability to obtain an effect equal to or more extreme than the one observed presuming the null hypothesis of no effect is true.
In other words, it is the probability of the data given the null hypothesis. However, it is often misunderstood to be the probability of the hypothesis given the data. The editors understand that the journey from data to hypothesis is a statistical inference, and one that in practice has turned out to be more misleading than informative. It encourages lazy thinking – if you reach the magical p-value then your hypothesis is true. They write:
In the NHSTP, the problem is in traversing the distance from the probability of the finding, given the null hypothesis, to the probability of the null hypothesis, given the finding. Regarding confidence intervals, the problem is that, for example, a 95% confidence interval does not indicate that the parameter of interest has a 95% probability of being within the interval. Rather, it means merely that if an infinite number of samples were taken and confidence intervals computed, 95% of the confidence intervals would capture the population parameter. Analogous to how the NHSTP fails to provide the probability of the null hypothesis, which is needed to provide a strong case for rejecting it, confidence intervals do not provide a strong case for concluding that the population parameter of interest is likely to be within the stated interval.
Another problem with the p-value is that it is not highly replicable. This is demonstrated nicely by Geoff Cumming as illustrated with a video. He shows, using computer simulation, that if one study achieves a p-value of 0.05, this does not predict that an exact replication will also yield the same p-value. Using the p-value as the final arbiter of whether or not to accept or reject the null hypothesis is therefore highly unreliable.
Cumming calls this the “dance of the p-value,” because, as you can see in his video, when you repeat a virtual experiment with a phenomenon of known size, the p-values that result from the data collection dance all over the place.
Regina Nuzzo, writing in Nature in 2014, echoes these concerns. She points out that if an experiment results in a p-value of 0.01, the probability of an exact replication also achieving a p-value of 0.01 (this all assumes perfect methodology and no cheating) is 50%, not 99% as many might falsely assume.
The real world problem is worse than these pure statistics would suggest, because of a phenomenon known as p-hacking. In 2011 Simmons et al. published a paper in Psychological Science in which they demonstrate that exploiting common researcher degrees of freedom could easily manipulate the data (even innocently) to achieve the threshold p-value of 0.05. They point out that published p-values cluster suspiciously around this 0.05 level, suggesting that some degree of p-hacking is going on.
This is also often described as torturing the data until it confesses. In a 2009 systematic review, 33.7% of scientists surveyed admitted to engaging in questionable research practices – such as those that result in p-hacking. The temptation is simply too great, and the rationalizations too easy – I’ll just keep collecting data until it wanders randomly over the 0.05 p-value level, and then stop. One might argue that overreliance on the p-value as a gold standard of what is publishable encourages p-hacking.
So what’s the alternative? Many authors here have suggested either doing away with the p-value, or (a less radical solution) simply bring it back down to its proper role – it provides one measure of the robustness of the data, but is not the final arbiter of whether or not the null hypothesis should be rejected. We have also supported those researchers who have called for increased use of Bayesian analysis as a more appropriate alternative. The Bayesian approach is to ask the right question, what is the probability of the hypothesis given both the prior probability and the new data?
The BASP give a lukewarm acceptance of the Bayesian approach:
Bayesian procedures are more interesting. The usual problem with Bayesian procedures is that they depend on some sort of Laplacian assumption to generate numbers where none exist. The Laplacian assumption is that when in a state of ignorance, the researcher should assign an equal probability to each possibility. The problems are well documented. However, there have been Bayesian proposals that at least somewhat circumvent the Laplacian assumption, and there might even be cases where there are strong grounds for assuming that the numbers really are there (see Fisher, 1973, for an example). Consequently, with respect to Bayesian procedures, we reserve the right to make case-by-case judgments, and thus Bayesian procedures are neither required nor banned from BASP.
OK – case-by-case analysis. That seems reasonable.
The journal editors are clear that their new policy does not mean they will accept less-than-rigorous research. They believe it will lead to more rigorous research:
However, BASP will require strong descriptive statistics, including effect sizes. We also encourage the presentation of frequency or distributional data when this is feasible. Finally, we encourage the use of larger sample sizes than is typical in much psychology research, because as the sample size increases, descriptive statistics become increasingly stable and sampling error is less of a problem.
Conclusion
I don’t know if the BASP solution to the problem of p-values is the best, ultimate, or only solution. Other solutions might include supplementing p-values with a discussion of the statistics that place them in their proper context, supplementing with Bayesian analysis, and having other requirements for scientific rigor. This would be a more difficult approach, and may not be able to dislodge the p-value from its lofty perch the way an outright ban might.
Requiring larger sample sizes is a good thing overall, but can create problems for young researchers just looking for a preliminary test of their new ideas. This then dovetails with another problem I and others have pointed out – presenting preliminary findings in the mainstream media as if they are definitive. Preliminary research is important, and if properly used can inform later research, but should not be used as a basis for clinical practice or hyperbolic headlines that ultimately misinform the public.
One solution is for journals to obviously separate preliminary research from confirmatory research. Preliminary research should be labeled as such with all the proper disclaimers and should not be the basis of hyped press releases. This may also provide the opportunity for having separate publishing rules for preliminary and confirmatory research – for example, for preliminary research journals can allow the use of p-values and techniques specifically designed to allow for smaller sample sizes.
The new BASP policy is a step in the right direction. At the very least I hope it raises awareness of the problems with relying on p-values and encourages a more nuanced understanding among researchers of statistics and methodological rigor.

Prevagen goes P-hacking
Can post-hoc data-dredging produce competent and reliable scientific evidence for Prevagen’s claims of memory improvement? The FTC and consumer groups say “no.”
Jann Bellamy on June 21, 2018
Link

Can a protein originally found in jellyfish improve your memory?
According to ubiquitous Prevagen ads,
Our Scientists Say “Yes!”
But, according the Federal Trade Commission (FTC), the state of New York, and consumer groups, what Prevangen’s scientists actually said was “No!” Hence a lawsuit filed last year by the FTC, joined by the New York Attorney General, against Quincy Bioscience (Quincy), the maker of Prevagen, for making false or misleading statements in labeling and advertisements.
The plaintiffs lost in federal district court, a decision that is now on appeal. How the case is ultimately decided could either strengthen the FTC’s hand in pursuing dietary supplement companies with flimsy evidence for their efficacy claims or encourage companies to go data-dredging when their studies don’t yield the desired results.
Oh no! Not another Prevagen ad!
If you are a sentient being with a cable subscription, you’ve no doubt seen advertisements for Prevagen and their claim that:
In clinical trials, Prevagen has been shown to improve short-term memory.
In one version, the results are illustrated with a blue jellyfish-themed chart showing a straight shot to memory improvement in 90 days.
The TV ads are one part of an aggressive marketing campaign which includes a website, social media, radio, print ads, infomercials (“The Better Memory Show”) and a “Prevagen Express” bus that visited health food stores and health expos. By all accounts, the company’s efforts have been wildly successful, with $165 million in total sales. A month’s supply of Prevagen costs between $24 and $68, depending on the formulation and source, and is widely available at retail outlets like CVS, Walgreens and Duane Reed, as well as through online sellers, like Amazon, and healthcare practitioners.
Prevagen is not the only player in the brain supplement business. According to Newsweek:
That often-desperate pursuit of remedies has created a lucrative marketing opportunity, and the supplement industry is cashing in. Products aimed at consumers worried about brain health and memory have contributed to a more than 10-fold increase in the number of supplements marketed in the U.S. over the last two decades.
At the request of Sen. Claire McCaskill, the Government Accountability Office investigated the marketing of brain and memory supplements and issued a report last year. The GAO noted that dietary supplements claiming to improve memory had sales estimated at $643 million in 2015, almost double the 2006 sales. Its report found consumer confusion regarding the roles of the FDA and FTC in regulating memory supplements, which could affect the ability of consumers to make complaints, and suggested better coordination between the two agencies.
Prevagen is a dietary supplement containing a synthetic version of the active ingredient, apoaequorin, a protein. Apoaequorin was originally obtained from a species of jellyfish called Aequorea victoria, which, say scientists, is not nearly as glamorous as its blue-glowing photographs suggest.
According to Quincy:
Apoaequorin has been shown in laboratory studies to support neuronal cells. Based on in vitro and in vivo animal studies, we hypothesized that apoaequorin has the potential to enhance memory and cognitive function in humans. Previous work with apoaequorin in aged canines demonstrated cognitive enhancement. [Citations omitted.]
And:
In a double-blinded, placebo-controlled trial, Prevagen demonstrated the ability to improve aspects of cognitive function in participants with either normal cognitive aging or very mild impairment, as determined by pre-trial screening.
Well, not exactly, says the FTC, as we’ll see in a moment.
Quincy also claims that Prevagen has undergone “extensive” safety testing and that no adverse effects were recorded when tested at dosage levels much higher than recommended, but does not reveal (except in fuzzy photos of the journal articles) that these were studies in rodents and petri dishes, not humans.
The feds step in
Prevagen caught the eye of regulators as early as 2012, when the FDA sent the company a warning letter stating that Prevagen was both an unproved drug and not legally a dietary supplement. At the time, Prevagen was marketed as a treatment for Alzheimer’s disease and head injury, claims only FDA-approved drugs can make. In addition, the FDA pointed out, apoaequorin is not a “dietary ingredient” as defined by the Dietary Supplement Health and Education Act, so Prevagen could not legally be marketed as a dietary supplement either. The FDA’s inspectors also found that Quincy had reported only two of more than 1,000 adverse events, some requiring hospitalization, and product complaints, including heart arrhythmias, chest pain, vertigo, tremors, seizures, strokes, and worsening of multiple sclerosis.
The current status of the FDA matter is unclear, but since the warning letter, Quincy appears to have stopped explicitly touting Prevagen for diseases, focusing instead on its claim that clinical trials show Prevagen improves memory. It is this claim that put the company in the sights of the FTC and the New York AG.
Last year, the FTC, which had received a number of consumer complaints about Prevagen, and the New York Attorney General, filed suit against Quincy, alleging violations of federal and state consumer protection laws, charging them with making false and unsubstantiated claims that Prevagen improves memory, provides cognitive benefits, and is “clinically shown” to work. According to an FTC spokesperson:
The marketers of Prevagen preyed on the fears of older consumers experiencing age-related memory loss. . . . But one critical thing these marketers forgot is that their claims need to be backed up by real scientific evidence.
The complaint sought a permanent injunction against the allegedly false and misleading claims made by Quincy, refunds for consumers, and “disgorgement of ill-gotten gains,” i.e., turning over profits made from Prevagen.
At the heart of the FTC’s complaint is the Madison Memory Study, paid for and conducted by Quincy, on which the company primarily relies in making claims that Prevagen is “clinically proven” to improve memory. The trial methodology and results have never been peer-reviewed or published in a scientific journal. The only written report available to the public is a “Clinical Trial Synopsis” on the company’s website.
The FTC and Quincy agree that the study was a double-blind, placebo-controlled human clinical study using objective outcome measures of cognitive function. It involved 218 subjects taking either 10 milligrams of Prevagen or a placebo, without a no-treatment group. The subjects were tested on nine computerized cognitive tasks to assess a variety of cognitive skills, including memory and learning, at intervals over 90 days. They also agree that the Madison Memory Study failed to show a statistically significant improvement in the treatment group over the placebo group on any of the nine cognitive tasks.
Unsurprisingly, to the FTC this means, contrary to advertisements, the “double-blinded, placebo controlled trial” of Prevagen did not “demonstrate the ability to improve aspects of cognitive function.” Thus, the FTC charges: (1) there is not “competent and reliable scientific evidence” to support Prevagen’s claims, which is required of health care product advertising, and (2) the claim that Prevagen is “clinically proven” is false.
Not so fast, says Quincy. That’s because, after failing to find a treatment effect for their study population as a whole, their researchers conducted more than 30 post hoc analyses of the results, looking at data broken down by several variations of smaller subgroups for each of the nine cognitive tasks. And here they found that older treatment subgroups having low to mild initial impairment had modest levels of improvement in three of the nine cognitive tasks.
In other words, Quincy went p-hacking, and those results, they say, are sufficient to support their advertising claims.
The FTC disagrees, stating in its complaint:
This methodology greatly increases the probability that some statistically significant differences would occur by chance alone. Even so, the vast majority of these post hoc comparisons failed to show statistical significance between the treatment and placebo groups. Given the sheer number of comparisons run and the fact that they were post hoc, the few positive findings on isolated tasks for small subgroups of the study population do not provide reliable evidence of a treatment effect.
The FTC points to the blue jellyfish-themed chart, reprinted in color in the complaint, as an example of how Quincy Bioscience is fudging with the Madison Memory Study results:
It indicates that a “double blinded, placebo controlled study” showed dramatic improvement in recall tasks when, in fact, the results for the specific task referenced in the chart showed no statistically significant improvement in subjects taking Prevagen compared to subjects taking a placebo. In addition, Defendants eliminated from the chart one of the four data points in the study – day 60. At day 60, the recall task scores of subjects taking Prevagen declined from day 30, and were slightly worse than the recall task scores of subjects in the placebo group.
The FTC also takes issue with Quincy’s assertion that apoaequorin is capable of crossing the blood-brain barrier, which, according to the FTC, is essential to their claim that their product improves memory and cognition. They don’t have evidence for this, according to the complaint, and, in fact, their own safety studies show that the protein is rapidly digested in the stomach and broken down into amino acids and small peptides, like other dietary proteins.
Prevagen defends its position in a brief filed with the appellate court as appropriate and common in the dietary supplement industry. In any event, it says, there is no law requiring the level of substantiation for healthcare product claims the FTC complains it failed to meet. Rather, the FTC itself has said it will take a flexible approach to substantiation, including the acceptance of less than “gold-standard” RCT studies.
It also argues, somewhat disingenuously, that it never said apoaequorin was capable of crossing the human blood-brain barrier. Rather, based on its canine studies, its:
marketing statement thus included only an indication “that apoaequorin is capable of crossing the blood brain barrier…in dog[s.]”
Of course, unless consumers were contemplating giving Prevagen to their dogs, this wouldn’t be relevant, unless one assumes this information was included so consumers would, by implication, conclude that apoaequorin can get into their brains as well. As the FTC says:
The first step in evaluating the truthfulness and accuracy of [dietary supplement] advertising is to identify all express and implied claims an ad conveys to consumers. Advertisers must make sure that whatever they say expressly in an ad is accurate. Often, however, an ad conveys other claims beyond those expressly stated. Under FTC law, an advertiser is equally responsible for the accuracy of claims suggested or implied by the ad. Advertisers cannot suggest claims that they could not make directly.
Ongoing litigation
Quincy successfully moved to dismiss the complaint for failure to state a claim. The district court ruled that the FTC, in its allegation that the post hoc analyses were insufficient to support Prevagen’s efficacy claims:
fails to do more than point to possible sources of error but cannot allege that any actual errors occurred.
As to the FTC’s claim that apoaequorin is rapidly digested and broken down into amino acids and small peptides like any other dietary protein, the court said:
This point, contradicted by canine studies whose relevance plaintiffs challenge, loses force when applied to the results of the subgroup study which make it clear that something caused a statistically significant difference between those subjects who took Prevagen and those given a placebo.

That, of course, was the whole point of the FTC’s first argument: post-hoc data-dredging is unreliable and can’t rule out “statistically significant difference” as mere noise.
On appeal, there is a substantial argument over whether a ruling on these issues was appropriate on a motion to dismiss, where the only question should be whether the allegations of the complaint are sufficient to state a claim against the defendants. Motions to dismiss are decided at the very earliest stages of a trial and only the plaintiffs’ complaint and defendants’ answer are under consideration. Courts are not supposed to consider any evidence or rule on questions of fact, two errors that the FTC says the district court made in this case. Whatever the merits of the case, I think that the complaint should not have been dismissed, and at least one commenter, also not a party to the litigation, agrees that the Second Circuit Court of Appeals is likely to reverse on this issue, giving the parties an opportunity to more fully flesh out the evidence and their arguments.
Joining, as amici curiae (“friends of the court”), the FTC and the state of New York in arguing the district court’s decision should be overturned is an impressive array of consumer groups and academics: Truth in Advertising, Inc., AARP, Advertising Law Academics, National Consumers League (their collective brief is here), Public Citizen, Center for Science in the Public Interest, and the Collaboration for Research Integrity and Transparency (collective brief here).
I had not heard of the Collaboration for Research Integrity and Transparency and found its mission interesting, and worthy:
The Collaboration for Research Integrity and Transparency (CRIT) is a multidisciplinary initiative of Yale Law School, Yale Medical School, and Yale School of Public Health. CRIT’s mission is to promote public health by improving the transparency and integrity of biomedical and clinical data. CRIT’s scientists have conducted research showing that data transparency and integrity are crucial to the accurate and informed use of drugs, devices, and biologics. Through litigation and policy work, CRIT focuses on enforcement of statutes and rules governing the accurate reporting of clinical trial results.
The latter amici curiae group explains in its brief why post-hoc analyses of data like that used by Quincy is not reliable. In part, they say:
Because the experts in the field of medical research would not generally accept Quincy’s post hoc subgroup analyses as yielding scientifically sound and reliable results, the FTC has adequately alleged that the analyses do not substantiate the claims and properly stated a claim on which relief can be granted. Subgroup analysis refers to an evaluation of study results in a subgroup of the subjects defined by certain baseline characteristics. See Rui Wang, et al., Statistics in Medicine – Reporting of Subgroup Analyses in Clinical Trials, 357 New Eng. J. of Med. 2189 (2007). Subgroup analyses are post hoc when the subgroup levels and “the hypotheses being tested are not specified before any examination of the data.” Id. at 2190. Post hoc subgroup analyses have long been viewed with skepticism by research scientists and statisticians. See, e.g., Andrew Oxman, et al., A Consumer’s Guide to Subgroup Analyses, 116 Annals of Internal Medicine 78, 83 (1992) (stating that “there are those who ignore scientific principles in the subgroup analyses they undertake and report, go on fishing expeditions, and indulge in data-dredging exercises” and that subgroup analyses showing small, marginally significant interactions generated by post hoc exploration of a single dataset “should be viewed with great skepticism”).
As another commenter pointed out, the FDA prohibits post-hoc subgroup analyses for efficacy claims in clinical drug trials, and explains why:
The FDA is concerned both about conflicts of interest and that performing multiple subgroup analyses can increase the risk of false-positive findings. In other words, the more analyses one conducts, the more likely it is that one will find a false positive result because when groups get smaller, confidence intervals grow larger and statistical power is reduced.
If the district court’s dismissal is overturned and the case remanded for further proceedings on the merits, the FTC will be able to present this scientific consensus to the court as evidence. (Even if it isn’t, the FTC and the New York AG remain free to file an amended complaint addressing the errors the district court in their pleadings.)
As well, if the case returns to the district court, the plaintiffs/appellants will be in good company in arguing the blood-brain barrier issue. Here’s what experts have said about that:
Meanwhile, Quincy’s claims are rejected by the research community, which notes that any protein taken by mouth will be broken down into amino acids in the stomach and won’t reach the brain in its original form.
Quincy’s claims “have no legitimate reality based on the real facts of science, nutrition and memory,” says David Mead, a molecular biologist and cofounder of Varigen Biosciences in Madison.
“It’s quackery. I can’t think of anything better to say,” says Baron Chanda, an associate professor in the UW-Madison Department of Neuroscience.
“I second the ‘quackery,'” says Chanda’s neuroscience colleague, professor Edwin Chapman. “This is downright silly — no other word for it.”
And:
“It is biologically inconceivable that taking a protein by mouth would have any effect on memory,” [Dr. David S.] Seres, [director of medical nutrition at Columbia University Medical Center] concluded.
For a longer, wonkier explanation of why apoaequorin is highly unlikely to cross the blood-brain barrier, see this Medscape article.
The importance of this case to the dietary supplement industry is evidenced by the fact that several industry trade groups have joined Quincy as amici curiae: Council for Responsible Nutrition, Consumer Healthcare Products Association, Natural Products Association, and the Alliance for Natural Health – USA.
My guess is that the lower court’s decision will be reversed and the case will go back to the district court, where this case will become another in the long history of expensive and time-consuming litigation over dietary supplements, much of which could be avoided if Congress ditched DSHEA and passed a law regulating dietary supplements and their marketing meant to protect consumers, not the industry. Here’s another prediction: that’s not going to happen.

Statistical Significance and Toxicity
Researchers propose to get rid of the use statistical significance in science reporting. The idea has merit.
Steven Novella on March 27, 2019
Link

Recently a study was published in The BMJ that purports to show a link between ambient pesticides (essentially mothers living near a farm that uses pesticides) and the risk of autism. This is feeding the anti-glyphosate frenzy that is currently in vogue, but it’s actually a good example of how slippery research methods result in dubious findings that get completely misreported.
Meanwhile, a commentary was also recently published in Nature essentially arguing that the concept of statistical significance should be abandoned. I discussed the paper at length on my other blog, but here is the quick version.
The primary method for determining significance is the p-value – a measure of the probability that the results obtained would deviate as much as they do or more from a null result if the null hypothesis were true. This is not the same as the probability that the hypothesis is false, but it is often treated that way. Also, studies often assign a cutoff for “significance” (usually a p-value of 0.05) and if the p-value is equal to or less than the cutoff the results are considered significant, if not then the study is negative.
There are multiple problems with the way in which the p-value and significance have come to be used in modern science and especially the reporting of science to the public. First, the p-value was never intended to be the one measure of how interesting scientific results are. A low p-value does not prove the hypothesis, and can occur even if the hypothesis being tested is false. Using a cutoff for significance is also a false dichotomy – in reality p-values are a continuum. There are many other aspects of the data that are as or more important than p-values, such as effect sizes, reproducibility, confidence intervals, and a Bayesian analysis of probability.
Relying on an arbitrary cutoff for significance also encourages p-hacking – exploiting researcher degrees of freedom to nudge the results over the finish line. P-values themselves are not very reproducible. If you do the identical study multiple times, you will get a range of p-values, so which is the correct one? In the end, p-values alone don’t tell us very much about the reality of the hypothesis.
In response to increasing awareness of the many problems with p-values and the overreliance on arbitrary “significance” several fixes have been proposed. Some journals have simply banned p-values. They will not publish studies that rely on them, and encourage the use of more thorough statistical analysis. Others have proposed to lower the cutoff for significance, from 0.05 to 0.005 for example, in order to at least reduce the false positives (although this would increase the false negatives).
This new commentary proposes yet another solution – keep the p-value, but get rid of an arbitrary cutoff for statistical significance. In its place, add a more thorough statistical analysis.
What does all of this wonky statistical arguing mean for the average lay person who just wants to know if eating chocolate will help them lose weight (it won’t)? The big take-away is – do not equate statistical significance with the hypothesis being true. Realize that most such studies are ultimately wrong. Wait for an expert to put the results into context. Think about things like plausibility – do the results even make sense?
With all this in mind let’s get back to the pesticide and autism study. What they did was gather data from registrars regarding use of pesticides and also autism diagnoses and did some fancy shmancy statistical analysis (if you must know, they did a multivariable logistic regression analysis). They found:
Risk of autism spectrum disorder was associated with prenatal exposure to glyphosate (odds ratio 1.16, 95% confidence interval 1.06 to 1.27), chlorpyrifos (1.13, 1.05 to 1.23), diazinon (1.11, 1.01 to 1.21), malathion (1.11, 1.01 to 1.22), avermectin (1.12, 1.04 to 1.22), and permethrin (1.10, 1.01 to 1.20). For autism spectrum disorder with intellectual disability, estimated odds ratios were higher (by about 30%) for prenatal exposure to glyphosate (1.33, 1.05 to 1.69), chlorpyrifos (1.27, 1.04 to 1.56), diazinon (1.41, 1.15 to 1.73), permethrin (1.46, 1.20 to 1.78), methyl bromide (1.33, 1.07 to 1.64), and myclobutanil (1.32, 1.09 to 1.60); exposure in the first year of life increased the odds for the disorder with comorbid intellectual disability by up to 50% for some pesticide substances.
Mainstream media reports boiled all this down to – glyphosate causes autism. But there are numerous caveats with this study, and they add up to the fact that we cannot conclude much from this data. The first thing I noticed was that the odds ratios were pretty tiny, so the effect size here is relatively small. Given that this is a population-based observational study, that means that there are many potential confounding factors. The authors try to control for the obvious ones they can think of, but it’s impossible to think of them all. Also, the smaller the effect size, the more subtle the confounding factor and the harder it is to control for.
However – there is a bigger problem which brings the statistical results themselves into question. When looking at the data another question that quickly came to mind was – how many different comparisons did they actually make? Well, the first response to the study published by The BMJ had the answer. John Tucker, PhD noted:
In a textbook example of multiple hypothesis testing , the authors examined the effects of estimated exposure to 11 different pesticides during 3 different developmental periods against two different adverse developmental outcomes. From among the 66 evaluated endpoints, they conclude that prenatal exposure to 6 of these pesticides is associated with 10-20% increases in risk of autism disorder, and that prenatal exposure to a partially overlapping list (3 of 6) is associated with autism disorder with intellectual disability.
This is one of the “researcher degrees of freedom” that Simmons et. al. warned about in their seminal paper. This is also extremely common in scientific papers – researchers look at lots of data from many different angles until they find something interesting, and then publish that. This is OK if you are just generating hypotheses in a preliminary study, but the data actually means nothing until you confirm the results with fresh data. In fact, there is increasingly a call for researchers to more routinely do just that. Don’t bother publishing until you have done an “internal replication”. Otherwise, we just flood the scientific literature with false positives from mining data for any coincidental correlations.
So in the end we have a paper that used multiple comparisons to find very small effect sizes, which is highly likely to be a spurious finding that will not replicate. But it doesn’t matter, because the ideologues have already seized upon this paper as vindication for the evils of glyphosate and even GMOs (even though this has nothing to do with GMOs). Explaining why the results are likely not reliable mostly causes people’s eyes to glaze over. (People have told me that as soon as I mention p-values they tune out.)
Clearly these statistical issues need to be sorted out by scientists and statisticians, and not the public at large or even the media. At least the problem is recognized and being discussed, and potential solutions being offered. Clearly the institutions of science need to take a thorough look at the whole issue of p-values and statistical significance and how they are used and reported in research.

5 out of 4 Americans Do Not Understand Statistics
Mark Crislip on December 13, 2013
Link
Ed: Doctors say he’s got a 50/50 chance at living.Frank: Well there’s only a 10% chance of thatNaked Gun
There are several motivations for choosing a topic about which to write. One is to educate others about a topic about which I am expert. Another motivation is amusement; some posts I write solely for the glee I experience in deconstructing a particular piece of nonsense. Another motivation, and the one behind this entry, is to educate me.
I hope that the process of writing this entry will help me to better understand a topic with which I have always had difficulties: statistics. I took, and promptly dropped, statistics 4 times a college. Once they got past the bell shaped curve derived from flipping a coin I just could not wrap my head around the concepts presented. I think the odds are against me, but I am going to attempt, and likely fail, in discussing some aspects of statistics that I want to understand better. Or, as is more likely, learn for the umpteenth time, only to be forgotten or confused in the future.
Frequentist
In medicine it is the p <= 0.05 that rules. If the results of the study meet that requirement the results are statistically significant. Maybe not clinically relevant or even true, but you have to have pretty good reasons not to bow before the power of a p <= 0.05. It is SIGNIFICANT, dammit.
But what does that mean? First you have to consider the null hypothesis: that two events are totally unrelated, that there is no difference between the two treatments in terms of their effect. The p value is the likelihood that any observed difference away from the null hypothesis is due to chance. Or
The probability of the observed result, plus more extreme results, if the null hypothesis were true.
So if there is a small p value, 0.05, then the chance that a difference observed in two treatments is random is 5%. If the p-value is ≤ 0.05 then the result is significant and the null hypothesis may be rejected and the alternative hypothesis, that there is a difference in the treatments, might be accepted.
The cut-off of significance, 0.05, it is should be emphasized, it an arbitrary boundary that was established by Fischer in 1925 but has since become dogma set in rebar-reinforced cement.
And what is significant, at least as initially formulated by Fischer?
Personally, the writer prefers to set a low standard of significance at the 5 percent point … A scientific fact should be regarded as experimentally established only if a properly designed experiment rarely fails to give this level of significance.
In other words, the operational meaning of a P value less than .05 was merely that one should repeat the experiment. If subsequent studies also yielded significant P values, one could conclude that the observed effects were unlikely to be the result of chance alone. So “significance” is merely that: worthy of attention in the form of meriting more experimentation, but not proof in itself.
The p value has numerous problems as a method for determining whether the null hypothesis can be rejected. There are at least 12 misconceptions with p value, all of which are common and I would have though true once upon a time:
1. If P = .05, the null hypothesis has only a 5% chance of being true.2. A nonsigniﬁcant difference (eg, P ≥.05) means there is no difference between groups.3. A statistically signiﬁcant ﬁnding is clinically important.4. Studies with P values on opposite sides of .05 are conﬂicting.5. Studies with the same P value provide the same evidence against the null hypothesis.6. P =.05 means that we have observed data that would occur only 5% of the time under the null hypothesis.7. P =.05 and P ≤ .05 mean the same thing.8. P values are properly written as inequalities (eg, “P ≤.02” when P = .015)9. P = .05 means that if you reject the null hypothesis, the probability of a type I error is only 5%.10. With a P = .05 threshold for signiﬁcance, the chance of a type I error will be 5%.11. You should use a one-sided P value when you don’t care about a result in one direction, or a difference in that direction is impossible.12. A scientiﬁc conclusion or treatment policy should be based on whether or not the P value is signiﬁcant.
My head is already starting to hurt. It appears from reading writers wiser than I that the p value is a piss-poor criterion to judge biomedical results.
Above all what the p value does not include is measure of the quality of the study. If there is garbage in, there will be garbage out. I would wager that the most popular way to find a significant p value is subgroup analysis, the Xigris study perhaps being the most expensive example of that bad habit. Just this week I was reading an article on high dose oseltamivir for treatment of influenza and
Subanalysis of influenza B patients showed faster RNA decline rate (analysis of variance, F = 4.14; P = .05) and clearance (day 5, 80.0% vs 57.1%) with higher-dose treatment.
And I have no end of colleagues who will see that meaningless p value and up the dose of the oseltamivir. Same as it ever was. To my mind most p of 0.05 is almost certainly random noise and clinically irrelevant.
The most important foundational issue to appreciate is that there is no number generated by standard methods that tells us the probability that a given conclusion is right or wrong. The determinants of the truth of a knowledge claim lie in combination of evidence both within and outside a given experiment, including the plausibility and evidential support of the proposed underlying mechanism. If that mechanism is unlikely, as with homeopathy or perhaps intercessory prayer, a low P value is not going to make a treatment based on that mechanism plausible. It is a very rare single experiment that establishes proof. That recognition alone prevents many of the worst uses and abuses of the P value. The second principle is that the size of an effect matters, and that the entire confidence interval should be considered as an experiment’s result, more so than the P value or even the effect estimate.
So what’s a science-based medical practitioner to do?
Bayes
If comprehending a p value gives me a headache, Bayes gives me a migraine. Bayes is evidently a superior conceptual framework for determining whether a result is ‘true’ and has none of the flaws of the p value. But Bayes also lacks the simplicity of a simple number and I prefer a simple number, especially given the volume of papers I read. The P-value is a shortcut, and unfortunately an unreliable shortcut.
If you Google Bayes theorem you always get that formula, an expression of the concept that is both concise and, for a practicing clinician, imminently forgettable and impossible to apply without help. I have a Bayes calculator on my iPhone and I re-read blog entities on the topic over and over. It is still difficult to comprehend and apply, at least for me.
As a clinician, as someone who takes of sick people for a living, and not a scientist, how do I apply Bayes?
Simplistically how valid, how true, a result might be depends in part on the prior plausibility. In a world of false positives and false negatives, it is not always so simple to determine if a positive test result makes a diagnosis likely or a treatment effective. Many of the Bayes explanation sites use cancer screening as an example and I cannot retain that example any longer than while I read it.
The problem with Bayes, although superior to p values, is that of pretest plausibility. Often it seems people are pulling pretest plausibilty out of thin air. In the old days we would do V/Q scans to diagnosis pulmonary embolism (PE), not a great test, and there was always the issue of how to interpret the result based on whether you thought by risk factors that the patient was likely to have had pulmonary embolism. I always felt vaguely binary during the discussions. Either they had a PE or they didn’t. The pre-test probability didn’t matter.
But it does. And that is my problem.
I have found the Rx-Bayes program for iOS gives a nice visual for understanding of Bayes. Even a highly sensitive and specific test is worthless if the pretest probability is low. I deal with this most often with Lyme testing. There is virtually no Lyme in Oregon, so a positive test is so much more likely to be a false positive than represent the real deal. It is striking how high the pretest probability has to be before even a sensitive and specific test has good reliability. And most tests have only a middling sensitivity and specificity.
Synthesis
The p value is so much nicer as it gives a single number rather than a range of probabilities. It is interesting to see what happens when you apply Bayes to p values:
If one starts with a chance of no effect of 50%, a result with a minimum Bayes factor of 0.15 (corresponding to a P value of 0.05) can reduce confidence in the null hypothesis to no lower than 13%. The last row in each entry turns the calculation around, showing how low initial confidence in the null hypothesis must be to result in 5% confidence after seeing the data (that is, 95% confidence in a non-null effect). With a P value of 0.05 (Bayes factor = 0.15), the prior probability of the null hypothesis must be 26% or less to allow one to conclude with 95% confidence that the null hypothesis is false. This calculation is not meant to sanctify the number “95%” in the Bayesian approach but rather to show what happens when similar benchmarks are used in the two approaches.These tables show us what many researchers learn from experience and what statisticians have long known; that the weight of evidence against the null hypothesis is not nearly as strong as the magnitude of the P value suggests. This is the main reason that many Bayesian reanalyses of clinical trials conclude that the observed differences are not likely to be true
As I understand it as a clinician, the take home is that a p of 0.05 or even 0.01 maybe statistically significant, it is unlikely to mean the result is ‘true’, that you can reject the null hypothesis. In part this is due to the unfortunate fact than many clinical studies stink on ice.
Recently an article in PNAS, way over my head, discussed how Bayes, by way of the Bayes factor, and the p value can be related.
The Bayes factor is a method by which the subjective prior probability is removed:
The Bayes factor is a comparison of how well two hypotheses predict the data. The hypothesis that predicts the observed data better is the one that is said to have more evidence supporting it.
How do the two values compare? A p of 0.05 corresponds to a Bayes factor of 3 to 5, considered weak evidence. The take home is that p of 0.005 is probably a better value for significant and 0.001 for highly significant if you really want to reject the null hypothesis. Not quite the 5 sigma criteria that CERN used to find the Higgs boson, but better and more likely to be true (true meaning the null hypothesis is unlikely).
Wallowing in the medical literature these last 30 years I have been struck how studies wobble about the zero. Some studies show benefit, some do not, of a given intervention, all with slightly different designs but all with middling p values.
My big take home from the above is to consider an intervention effective, as true, if the p is 0.005 AND has been replicated. But a p of 0.05 in a single study? Pffffftttttttt.
I think the above analysis probably excludes a big chunk of real medicine and all the topics covered by this blog as being true. I wonder, is there an SCAM intervention that has a p of 0.005, much less 0.001? Not that I can find, but I am sure the comments will correct me about this and the numerous other errors I have made in the essay.
I think I have a simple rule of thumb with a sophisticated background. As Barbie noted, math is hard.

Pitfalls in Research: Why Studies Are More Often Wrong than Right
Here is a course guide to episode 9, “Pitfalls of Research”, of my YouTube lecture series on science-based medicine.
Harriet Hall on March 3, 2020
Link

Alternative medicine relies heavily on anecdotal evidence and personal experience, but it knows it won’t convince mainstream science unless it can provide scientific validation. It is quick to crow about positive results from scientific studies, often from studies in laboratory mice or Petri dishes, or from human studies that are poorly designed and may not even use a control group. Science-based medicine relies on evidence from clinical trials, especially those that are randomized, double-blind, and placebo-controlled. But that evidence can be misleading. The results of even the best studies are more likely to be wrong than right. Positive results from preliminary trials are all too often overturned by negative results from subsequent larger, better trials. Individual studies are often wrong, but that doesn’t mean we should ignore them. Science is a collaborative, self-correcting enterprise that can be relied on to come ever closer to reality over the course of time. A good rule of thumb is never to believe a single study, but to wait for replication, confirmation, and a consensus of all the available data. And to consider prior probability.
In my free YouTube video lecture course on science-based medicine, I talk about “Pitfalls in Research” in lecture 9. I have been asked to re-cycle the information in the course guide [PDF] for that lecture in the form of an SBM post because it provides valuable guidance for evaluating scientific studies.
Ioannidis showed that most research findings are false, especially in the case of:
    • Small studies
    • Small effect size
    • Multiple endpoints
    • Financial interests and bias
    • Hot topic with more teams in competition
    • Research on improbable areas of CAM
Things to Keep in Mind:	Because:
Media reports can’t be trusted	Reporters often don’t understand the science and merely regurgitate press releases
Learn what can go wrong	The same flaws keep repeating
Never believe one study	All data pro and con must be considered
Prior probability matters	Basic science can’t be ignored
Bausell’s quick test
In Snake Oil Science, Bausell provided this quick 4-point checklist:
     1.  Is the study randomized with a credible control group?
     2.  Are there at least 50 subjects per group?
     3.  Is the dropout rate 25% or less?
     4.  Was it published in a high-quality, prestigious, peer-reviewed journal?(You can look up the journal’s impact factor)
If the answers to these questions are all Yes, that doesn’t guarantee that the study’s conclusions are credible, but if some of the answers are No, it makes the results more questionable.
Other questions to ask
Was the study done on people or fruit flies?	Studies in animals and in vitro may not apply to humans.
Who are the subjects?	Are they representative? Could they be biased?
Who funded the study?	Could be a source of bias.
Could the authors have been biased?	Look for conflicts of interest.
Was randomization adequate?	Were the groups really comparable?
Was blinding adequate?	Was there an exit poll to see if they could guess whether they were in the test group or the control group?
Were there multiple endpoints? (This, and the next point, are known as “p-hacking“)	If so, were appropriate statistical corrections done?
Was there inappropriate data mining?	Sometimes the study is negative but the data are tortured until they seem to confess something positive.
Where was the study done?	Studies done in China are suspect because they never report negative results.
Were the results clinically meaningful?	Statistical significance ≠ clinical significance.
Did the results merit the conclusions?	Sometimes they don’t.
Were they studying something real?	Tooth Fairy science tries to study imaginary things like Tooth Fairies, acupoints, and human energy fields.
Was it a “pragmatic” real-world study?	Pragmatic studies are not meant to establish efficacy. An unproven treatment with strong placebo effects may appear to outperform a proven treatment.
Other things that might go wrong
    • Errors of math
    • Using the wrong statistical test for the kind of data collected
    • Contaminants
    • Poor compliance
    • Conscious or unconscious manipulation of data by technician
    • Procedures not carried out properly
    • Deliberate fraud
Important definitions to understand
Statistically significant	Significant at p 0.5 ≠ Truth. It just measures the chance a repeat study will get positive results if the null hypothesis is true (no difference between groups).
Clinically significant	Changes disease outcome
Relative risk	Reports percentages
Absolute risk	Reports actual numbers of people
Number Needed to Treat (NNT)	Number Needed to Treat for one person to benefit
Number Needed to Harm (NNH)	Number Needed to Treat for one person to be harmed.
Confidence intervals	We can be confident the true value is somewhere in this range.
Correlation ≠ causation	Correlation = a statistical relationship between 2 variables. Doesn’t mean one causes the other.
Sensitivity of a test	Percentage of people with the disease who test positive.
Specificity of a test	Percentage of people who don’t have the disease who test negative.
Base rate	The prevalence in the population. The rarer the condition, the more false positive test results.
Positive Predictive Value (PPV)	The likelihood that a positive test is true: that you actually have the disease.
Negative Predictive Value (NPV)	The likelihood that a negative test is true: that you don’t have the disease.
Problems that need correcting	Solutions
· Poor quality studies
· Publish or perish
· Publication bias
· Lack of replication
· Mistakes missed by peer review
· Pay-to-publish journals
· Big Pharma malfeasance	· Better education of researchers
· Quality control at journals
· Publish replications and negative studies
· Register all studies
· Full disclosure
· Improve media reporting
Science is far from perfect
John Rennie said, “There seems to be no study too fragmented, no hypothesis too trivial, no literature citation too biased or too egotistical, no design too warped, no methodology too bungled, no presentation of results too inaccurate, too obscure, and too contradictory, no analysis too self-serving, no argument too circular, no conclusions too trifling or too unjustified, and no grammar and syntax too offensive for a paper to end up in print.”
A warning
This warning statement should be attached to all research studies:“Warning! Taking any action on the basis of this research could result in injury or death. The results described in this study have not been replicated and the long-term effects of this treatment are unknown. Past performance is no guarantee of future results. When subjected to further investigation, most published research findings turn out to be false.”
Two more things to remember
     1.  My SkepDoc’s Rule: Never accept a claim until you understand who disagrees with it and why.
Don’t ever forget to ask if you yourself might be biased.

Information Literacy and the Number Needed to Treat
Steven Novella on November 4, 2015
Link

Increasingly people are accessing healthcare information in order to make decisions for their own health. A 2010 Pew poll found that 80% of internet users will do so for health care information. This presents a huge potential benefit, but also a significant risk.
Information literacy
Daniel Levitin talks about the need for public information literacy, something we also discuss frequently here on SBM. If you are accessing the internet to inform your health care decisions, then you need to know how to determine the legitimacy and trustworthiness of the websites you are visiting. There is a big difference between NaturalNews (a crank site full of misinformation and conspiracy theories) and Nature News (an outlet for one of the most prestigious science journals in the world).
Even when you can discriminate between good and bad health information websites, the challenge remains to properly interpret the scientific information to which you now have access.
Part of the problem is that the original source of scientific information is the published technical literature, which is designed for optimal communication among experts. It is extremely difficult to properly interpret a technical paper unless you have a fair degree of expertise in the field. Even as a physician it can be challenging for me to fully understand a research study in a medical field far from my specialty of neurology.
Most of the public, therefore, has to rely on experts to interpret that data for them and present it in a way that is digestible and meaningful. Properly communicating complex science to a non-expert general public is a skill set unto itself, and presents many challenges.
Unfortunately, often that interpretation is being done by non-experts, by journalists who many not even have a specialty in science journalism. Even worse, the information could be presented by biased, commercial, or agenda-driven sources.
As one example, often reporting of health information does not properly distinguish between relative risk and absolute risk. A headline might declare that a new treatment reduces the risk of a bad health outcome by 50%. That sounds impressive. But this is a relative risk reduction, and may reflect an absolute risk reduction of 0.2% to 0.1%, or an absolute risk reduction of 0.1% (that sounds less impressive).
Number needed to treat
This brings us to a specific method for representing the health effects of a treatment or the effectiveness of a diagnostic procedure – the number needed to treat (NNT). Daniel Levitin mentioned above advocates for the NNT as a way of communicating health information to the public, and he has a point.
There is also a group of physicians, the NNT group, who advocate for using the NNT to communicate health information. They have a useful website that translates healthcare statistics into NNT.
Put simply, the NNT is the number of patients who need to take a treatment in order for one person to receive a specific benefit. This may put statistical information into a context that makes it easier for the general public, and physicians also, to understand.
For example, a few years ago I wrote about a Cochrane review of statin drugs for primary prevention of heart disease. The review found:
What they found was that total mortality had a relative risk reduction of 17%, risk of heart attacks was reduced by 28%, and strokes by 22%.
If the information were only stated in this way, even identifying the numbers as relative risk, the average person might walk away thinking the effects are fairly significant. However, you could also represent the data by stating that the NNT was 1 in 1,000. The data show that for primary prevention you have to treat 1,000 patients for one year in order to prevent one death. This still may be worth is, as death is a pretty significant negative outcome, but it helps understand what the effect of the drugs truly is.
Levitin also uses statins as an example. One popular statin has an NNT to reduce cholesterol of 1 in 300 – only one person is 300 will actually have their cholesterol reduced by the drug. However, the number needed to harm is 20 – one person in 20 being treated will have side effects (muscle pain and cramping, for example).
For another example, let’s take a look at using warfarin instead of aspirin for atrial fibrillation in order to prevent strokes and blood clots. The NNT for strokes was 60, while the NNT for systemic blood clots was 360. Meanwhile, 1 in 167 had a fatal hemorrhage while 1 in 25 had a bleed requiring hospitalization. In other words, for every three strokes you prevent, you cause one death and put seven people in the hospital with a bleed.
The NNT is also a good way to look at cost effectiveness. This is unfortunately increasingly important as we try to rein in rising health care costs. No one wants to think in these terms, but it can be useful to consider how much it costs to treat those 300 people for one year and the 15 who will have side effects in order to reduce the cholesterol in one person. The NNT makes it a little more intuitive to see what we are getting for our health care dollars.
Conclusion: NNT provides useful context for medical decision-making
The NNT is a legitimate and highly useful way to present medical statistics. It is perhaps more intuitive to understand than relative or absolute risk reduction. It might be useful for medical journals to require scientists to provide NNT data for their studies. This would also make it easier for journalists and science communicators to provide the NNT as well.
The NNT is not new. It was first described in 1988. I found an editorial from 1997 recommending the adoption of NNT to help communicate health statistics. They concluded:
We advocate an approach to systematic reviews that distills information into, in effect, one number: the NNT. This is simple to remember and directly supports efforts to work with patients to make the best possible clinical decisions for their care.
Almost two decades later, the NNT is gaining ground but has not achieved widespread adoption. Habits are often slow to change. Perhaps the explosion of health care information online will provide a further incentive to adopt the NNT as a standard format for communicating health statistics to the public.

   The Value of Replication
Steven Novella
Link
Daryl Bem is a respected psychology researcher who decided to try his hand at parapsychology. Last year he published a series of studies in which he claimed evidence for precognition — for test subjects being influenced in their choices by future events. The studies were published in a peer-reviewed psychology journal, the Journal of Personality and Social Psychology. This created somewhat of a controversy, and was deemed by some to be a failure of peer-review.
While the study designs were clever (he simply reversed the direction of some standard psychology experiments, putting the influencing factor after the effect it was supposed to have), and the studies looked fine on paper, the research raised many red flags — particularly in Bem’s conclusions.
The episode has created the opportunity to debate some important aspects of the scientific literature. Eric-Jan Wagenmakers and others questioned the p-value approach to statistical analysis, arguing that it tends to over-call a positive result. They argue for a Bayesian analysis, and in their re-analysis of the Bem data they found the evidence for psi to be “weak to non-existent.” This is essentially the same approach to the data that we support as science-based medicine, and the Bem study is a good example of why. If the standard techniques are finding evidence for the impossible, then it is more likely that the techniques are flawed rather than the entire body of physical science is wrong.
Now another debate has been spawned by the same Bem research — that involving the role and value of exact replication. There have already been several attempts to replicate Bem’s research, with negative results: Galak and Nelson, Hadlaczky, and Circee, for example. Others, such as psychologist Richard Wiseman, have also replicated Bem’s research with negative results, but are running into trouble getting their studies published — and this is the crux of the new debate.
According to Wiseman, (as reported by The Psychologist, and discussed by Ben Goldacre) the Journal of Personality and Social Psychology turned down Wiseman’s submission on the grounds that they don’t publish replications, only “theory-advancing research.” In other words — strict replications are not of sufficient scientific value and interest to warrant space in their journal. Meanwhile other journals are reluctant to publish the replication because they feel the study should go in the journal that published the original research, which makes sense.
This episode illustrates potential problems with the scientific literature. We often advocate at SBM that individual studies can never be that reliable — rather, we need to look at the pattern of research in the entire literature. That means, however, understanding how the scientific literature operates and how that may create spurious artifactual patterns.
For example, I recently wrote about the so-called “decline effect” — a tendency for effect sizes to shrink or “decline” as research on a phenomenon progresses. In fact, this was first observed in the psi research, as the effect is very dramatic there — so far, all psi effects have declined to non-existence. The decline effect is likely a result of artifacts in the literature. Journals are more inclined to publish dramatic positive studies (“theory-advancing research”), and are less interested in boring replications, or in initially negative research. A journal is unlikely to put out a press release that says, “We had this idea, and it turned out to be wrong, so never-mind.” Also, as research techniques and questions are honed, research results are likely to become closer to actual effect sizes, which means the effect of researcher bias will be diminished.
If the literature itself is biased toward positive studies, and dramatic studies, then this would further tend to exaggerate apparent phenomena — whether it is the effectiveness of a new drug or the existence of anomalous cognition. If journals are reluctant to publish replications, that might “hide the decline” (to borrow an inflammatory phrase) — meaning that perhaps there is even more of a decline effect if we consider unpublished negative replications. In medicine this would be critical to know — are we basing some treatments on a spurious signal in the noise of research.
There have already been proposals to create a registry of studies, before they are even conducted (specifically for human research), so that the totality of evidence will be transparent and known — not just the headline-grabbing positive studies, or the ones that meet the desires of the researchers or those funding the research. This proposal is primarily to deal with the issue of publication bias — the tendency not to publish negative studies.
Wiseman now makes the same call for a registry of trials before they even begin to avoid the bias of not publishing replications. In fact, he has taken it upon himself to create a registry of attempted replications of Bem’s research.
While this may be a specific fix for replications for Bem’s psi research — the bigger issues remain. Goldacre argues that there are systemic problems with how information filters down to professionals and the public. Reporting is highly biased toward dramatic positive studies, while retractions, corrections, and failed replications are quiet voices lost in the wilderness of information.
Most readers will already understand the critical value of replication to the process of science. Individual studies are plagued by flaws and biases. Most preliminary studies turn out to be wrong in the long run. We can really only arrive at a confident conclusion when a research paradigm produces reliable results in different labs with different researchers. Replication allows for biases and systematic errors to average out. Only if a phenomenon is real should it reliably replicate.
Further — the excuse by journals that they don’t have the space now seems quaint and obsolete, in the age of digital publishing. The scientific publishing industry needs a bit of an overhaul, to fully adapt to the possibilities of the digital age and to use this as an opportunity to fix some endemic problems. For example, journals can publish just abstracts of certain papers with the full articles available only online. Journals can use the extra space made available by online publishing (whether online only or partially in print) to make dedicated room for negative studies and for exact replications (replications that also expand the research are easier to publish). Databases and reviews of such studies can also make it as easy to find and access negative studies and replications as it is the more dramatic studies that tend to grab headlines.
Conclusion
The scientific endeavor is now a victim of its own success, in that research is producing a tsunami of information. The modern challenge is to sort through this information in a systematic way so that we can find the real patterns in the evidence and reach reliable conclusions on specific questions. The present system has not fully adapted to this volume of information, and there remain obsolete practices that produce spurious apparent patterns in the research. These fake patterns of evidence tend to be biased toward the false positive — falsely concluding that there is an effect when there really isn’t — or at least in exaggerating effects.
These artifactual problems with the literature as a whole combine with the statistical flaws in relying on the p-value, which tends to over-call positive results as well. This problem can be fixed by moving to a more Bayesian approach (considering prior probability).
All of this is happening at a time when prior probability (scientific plausibility) is being given less attention than it should, in that highly implausible notions are being seriously entertained in the peer-reviewed literature. Bem’s psi research is an excellent example, but we deal with many other examples frequently at SBM, such as homeopathy and acupuncture. Current statistical methods and publication biases are not equipped to deal with the results of research into highly implausible claims. The result is an excess of false-positive studies in the literature — a residue that is then used to justify still more research into highly implausible ideas. These ideas can never quite reach the critical mass of evidence to be generally accepted as real, but they do generate enough noise to confuse the public and regulators, and to create an endless treadmill of still more research.
The bright spot is that highly implausible research has helped to highlight some of these flaws in the literature. Now all we have to do is fix them.

Outcome Bias in Clinical Decision Making and the Assessment of Our Peers
Clay Jones on December 5, 2014
Link

There are many mental pitfalls and logical stumbling blocks faced by healthcare professionals when attempting to untangle the complex web of patient history and physical exam findings. They can impede our ability to practice high quality medicine at every step in the process, interfering with our ability to establish an accurate diagnosis and to provide comfort or cure. And we are all susceptible, even the most intelligent and experienced among us. In fact, having more intelligence and experience may even enlarge our bias blind spots.
Steven Novella discussed the complexities of clinical decision making in early 2013, specifically tackling some of the more common ways that physicians can come to a faulty conclusion in the third installment of the series. One cognitive bias yet to be specifically addressed on the pages of Science-Based Medicine, and it is one that I encounter regularly in practice, is outcome bias. Simply put, outcome bias in medicine occurs when the assessment of the quality of a clinical decision, such as the ordering of a particular test or treatment, is affected by knowledge of the outcome of that decision. We are prone to assigning more positive significance to a decision when the outcome is positive, and we often react more harshly when the outcome is negative. This bias is particularly obvious when the result of a decision largely comes down to chance.
I see outcome bias rear its ugly head in two contexts for the most part: the Lucky Catch and the Bad Call.
The Lucky Catch
From Scrubs, season 8, episode 6, “My Cookie Pants”:
JD: I don’t know who you think you are. But I promise you, you are gonna regret this mistake for the rest of your – Nurse: Sorry to interrupt. Great call ordering that endoscopy, Dr. Mahoney. Mr. Lawton has stomach cancer.
JD: Jo, I’m sorry. I thought you were being callous with Mr. Lawton, and you were just being thorough.
Most doctors have their own personal lucky catch or “great call” story, or at least can easily recount one that was passed down to them at some point in their career. A patient presents with an unusual symptom or syndrome, and in a seeming flash of inspiration a particular lab or imaging modality is requested that reveals the rare or unlikely diagnosis. What tends to follow is some degree of awe at the clinical acumen of the ordering physician, and in some cases more than a bit of hindsight bias. (“Well of course that’s what the guy had! The clues were right in front of them the whole time.”)
In reality, medical mystery-type lucky catches like this are the exception rather than the norm. The much more common version, but one less likely to achieve legend status in a physician’s personal narrative, tends to occur when we’ve dusted off the diagnostic shotgun. Shotguns disperse multiple pellets in a wide pattern in order to increase the likelihood of making contact with the target. So does an ordering physician when they request a large number of tests thoughtlessly.
Sometimes, depending on the severity of the patient’s illness, the diagnostic shotgun is a necessary tool in our armamentarium. A critically ill patient unable to provide a personal medical history, for example, often deserves a more substantial initial work-up. A rapidly-approaching death may preclude a tiered approach to testing. More often than not, however, the shotgun approach is unhelpful and merely increases the chance of mistakenly assuming that a false positive finding is playing a role in the patient’s illness. It can also result in an incidental but real finding that exposes a patient to the risks of additional testing and treatment for something that may have never been a problem. And yes, sometimes a shotgun approach uncovers a surprising cause of the patient’s complaints.
Outcome bias kicks in when we look back at the decisions that occurred prior to the lucky catch or positive outcome and judge them more positively, even when the care in question was of poor quality. We often forget or even fail to acknowledge when a test or treatment had an unfavorable risk versus benefit ratio. This bias can reinforce the drive to perform more unnecessary testing in the future, not only by the directly-involved parties but also by any impressionable learners hearing the tale in the years to come.
The Bad Call
From Scrubs, season 5, episode 20, “My Lunch” (from 1:00):
JD: There’s no way you could have seen that coming. I mean, rabies? Come on, there’s, like, three reported cases a year. In fact testing for it would have been irresponsible. You would have wasted time those people didn’t have.
Dr. Cox: I was obsessed with getting those organs.
JD: You had to be. The fact is, those people were going to die in a number of hours and you had to make a call. I would have made the same call.
….
Dr. Cox: Aw, come on. Come on, Come on! GOD! GOD, GOD!
(Dr. Cox throws the paddles at the defibrillator and smashes the machine against the wall. Carla and the other nurses clean up and exit. Mr. Bradford’s body vanishes and Dr. Cox is left standing alone. J.D. enters.)
Dr. Cox: He wasn’t about to die, was he, Newbie? Could have waited another month for a kidney.
(Dr. Cox exits the room and removes his gloves.)
JD: Where are you going? Your shift’s not over. Hey! Remember what you told me? The second you start blaming yourself for people’s deaths…there’s no coming back.
Dr. Cox: (faces JD) Yeah…you’re right.
Okay, that was a bit melodramatic. It is a TV show after all. But outcome bias can also lead physicians to judge their own decisions, or the decisions of others, more harshly when a bad outcome occurs. It can even result in being less likely to recommend appropriate science-based tests or treatments in the future. When a pharmaceutical agent or surgery is indicated, even strongly recommended, it will never be without some degree of risk but it can be very difficult to separate the decision from the outcome in a rational way when a patient has suffered.
It’s easy to be hard on ourselves, but even easier to be hard on other physicians that we don’t have a personal relationship with. Jousting is a term that in a medical context occurs when one physician speaks poorly of another, or in some way implies that a decision made by another caregiver was inappropriate. It is very hard to avoid jousting because it often happens without us even realizing we are doing it. One doesn’t have to say that Dr. A is an idiot in a “Dr. House”-style rant to effectively undermine a fellow doctor and increase the risk of both parties being involved in malpractice litigation. Offhand comments can be just as effective. (“Oh, I’ve never heard of that approach to this condition.” or “I can’t believe something wasn’t done sooner.”)
Another cognitive hiccup that can add to the impact of outcome bias when evaluating the decisions of other healthcare professionals is the fundamental attribution error. When a negative outcome is known, we tend to immediately blame some internal factor in the involved physician – incompetence or intellectual laziness, for example – and we downplay or completely ignore situational or external factors that were out of their control and may have played a role. We are quick to blame a surgeon for a complication, for example, even when there may be a known rate for that complication that is not affected by procedural skill.
Conclusion
Some days it seems that the deck is stacked against us. Outcome bias and the fundamental attribution error are just two of many biases that can alter our perception of reality. In the case of outcome bias, it not only can affect how we judge our own actions but also the actions of others. And as if that weren’t enough, it has the potential to change how we practice medicine in the future because of our flawed assessments of prior decisions. As always, the key to combating errors in thinking like these is knowing that they exist in the first place. With that knowledge in place, when we are called upon to review the management performed by a peer, or reflect on our own decisions, these biases should be easier to reduce or maybe even avoid altogether.

   Freeways, autism, and correlation versus causation
David Gorski
Link
I have a love-hate relationship with epidemiology.
I love how epidemiology can look for correlations in huge sample sizes, sample sizes far larger than any that we could ever have access to in clinical trials, randomized or other. I love the ability of epidemiology to generate hypotheses that can be tested in the laboratory and then later in clinical trials. Also, let’s not forget that epidemiology is sometimes the only tool available to us that can answer some questions. Such questions generally involve hypotheses that can’t be tested in a randomized clinical trial because of either ethical concerns or others. A good example of this is the question of whether vaccines cause autism. For obvious ethical reasons, it’s not permissible to perform a randomized clinical trial in which one group of children is vaccinated and one is not, and then outcomes with respect to neurodevelopmental outcomes, such as autism and autism spectrum disorders, are tracked in the two groups. The ethical concern with such a study, of course, is the potential harm that would be likely to come to the unvaccinated control group, children who would be left unprotected against common and postentially deadly communicable diaseases.
On the other hand, epidemiology is one of the messiest of sciences, and epidemiological studies are among the most difficult in all of science to perform truly rigorously. The number of factors that can confound are truly amazing, and as a result, it’s very, very easy for an epidemiological study to detect apparent correlations that are either spurious or appear much stronger than the “true” correlation. There can be confounding factors beneath confounding factors wrapped in more confounding factors, the relationships among which are not always apparent. Not infrequently, a condition can appear to be correlated with, for instance, an environmental factor, but in reality that environmental factor and the condition both correlate with a third, unknown confounder. Worse, epidemiologists know that correlation does not necessarily equal causation, but the general public, for the most part, does not, which is why, when anti-vaccine activists, for instance, point out to a rising autism prevalence and then point out that autism prevalence started rising around the same time the vaccine schedule was expanded, to the average layperson the argument sounds compelling. As a result, the design of an epidemiological study is paramount in order to account for or minimize such factors. That’s why I always said I can’t be an epidemiologist. Even though I was very good at math in college, the statistics still made my brain hurt, and I don’t have the patience for the messiness of trying to account for all the possible confounding factors.
However, for all their strengths and flaws, epidemiological studies are an integral part of science-based medicine. They are used to identify predisposing factors to diseases and conditions, environmental contributors to disease, and adverse reactions to drugs, among many other useful pieces of data. That’s why, from time to time, I like to examine epidemiological studies, particularly if they’re epidemiological studies that are getting a lot of press.
The use and abuse of autism epidemiology studies
For instance, studies like this one described in a story in the Los Angeles Times on Friday entitled Proximity to freeways increases autism risk, study finds: More research is needed, but the report suggests air pollution could be a factor:
Children born to mothers who live close to freeways have twice the risk of autism, researchers reported Thursday. The study, its authors say, adds to evidence suggesting that certain environmental exposures could play a role in causing the disorder in some children.
“This study isn’t saying exposure to air pollution or exposure to traffic causes autism,” said Heather Volk, lead author of the paper and a researcher at the Saban Research Institute of Children’s Hospital Los Angeles. “But it could be one of the factors that are contributing to its increase.”
Another news story describes the study thusly:
Story link: MyFoxLA.com
Naturally, the anti-vaccine crank blog Age of Autism is expressing interest in this study. The interview, of course, is truly execrable, particularly the bubble-head host saying “some people” think it’s the MMR, as though the anti-vaccine movement who demonize the MMR as a cause of autism have the slightest clue what they are talking about, and mentioning Jenny McCarthy as though she were anything other than a bad actress with a Google University education that led her to believe that vaccines caused her son’s autism.
Before I get to the study itself, one thing I always wondered about is why anti-vaccine groups are so keen on buying every study that purports to show an environmental component that might predispose to autism. Naturally, they often manage to related it back to vaccines, as they tried to do when there was a bad study purporting to show a link between proximity to power plants whose emissions contained mercury and autism. At least with that study, the potential relationship was fairly obvious in that a large contingent of the anti-vaccine movement still blames mercury in the thimerosal used in childhood vaccines vaccines as a preservative (in the U.S., at least) until the end of 2001. True, the chemical form of mercury in the environment is different than thimerosal, with different implications for toxicity and metabolism, but to the anti-vaccine mercury is mercury is mercury, and it’s equally evil.
Personally, I suspect that the reason anti-vaccine groups glom onto any study that purports to identify an environmental contributor to autism is guilt by association. If, “reason” anti-vaccine activists, this pollution or pesticides are implicated as a cause of autism, then maybe vaccines do too, and they aren’t so crazy after all for saying so. I’m sure regular readers of this blog can see the flaws in that logic. Also, many anti-vaccine activists are also advocates of a wide variety of “autism biomed” quackery, which can range from chelation therapy, to supplements, to hyperbaric oxygen chambers, and even to stem cell quackery. If pesticides, chemicals, environmental pollutants, or whatever can be implicated as a cause or contributor to autism, each new “cause” opens up a new panoply of “autism biomed” quackery to be tried.
So let’s get to the meat of the study itself.
The CHARGE study, proximity to freeways, and autism
My usual loghorreic introduction complete, let’s take a look at the study itself. It’s by a group of investigators from the University of Southern California, and its lead investigator was Heather Volk, the woman who appeared on the above FOX News segment, and the title of the article is Residential Proximity to Freeways and Autism in the CHARGE Study. Conveniently, the anti-vaccine crew at Age of Autism has posted an actual PDF of the study. One of these days, they’re going to get into trouble for doing that, but for now it allows anyone who doubts my analysis to go straight to the source. Let’s take a look at the abstract, for those of you who don’t want to read the whole paper:
Background: Little is known about environmental causes and contributing factors for autism. Basic science and epidemiological research suggest that oxidative stress and inflammation may play a role in disease development. Traffic-related air pollution, a common exposure with established effects on these pathways, contains substances found to have adverse prenatal effects.
Objectives: To examine the association between autism and residence proximity, during pregnancy and near the time of delivery, to freeways and major roadways as a surrogate for air pollution exposure.
Methods: Data were from 304 autism cases and 259 typically developing controls enrolled in the Childhood Autism Risks from Genetics and the Environment (CHARGE) Study. The mother’s address recorded on the birth certificate and trimester specific addresses derived from a residential history obtained by questionnaire were geo-coded and measures of distance to freeways and major roads were calculated using ArcGIS software. Logistic regression models compared residential proximity to freeways and major roads for autism cases and typically developing controls.
Results: Adjusting for sociodemographic factors and maternal smoking, maternal residence at the time of delivery was more likely be near a freeway (≤309 meters) for cases, as compared to controls (odds ratio (OR), 1.86, 95% confidence interval (CI) 1.04-3.45). Autism was also associated with residential proximity to a freeway during the third trimester (OR, 2.22, CI, 1.16- 4.42). After adjustment for socio-economic and demographic characteristics, these associations were unchanged. Living near other major roads at birth was not associated with autism. Conclusions: Living near a freeway was associated with autism. Examination of associations with measured air pollutants is needed.
There are a number of problems with this study, as you might imagine. First off, in epidemiological terms, it’s rather small for a study of this nature. On the other hand, it does have the strength that each child was examined individually for various factors; in other words, individual-level data were used rather than ecological (group-level) data, the latter of which has a huge tendency to result in false positives. Worse, there are a number of inconsistencies in the results, but in particular one glaring confounding factor that apparently failed to be taken into account, at least as far as I can tell.
First, let’s look at the anomalous findings. The main finding of this study is that being within 309 m of a freeway was associated with a barely statistically significant odds ratio of being diagnosed with an autism spectrum disorder of 1.86 (95% confidence interval: 1.03 to 3.45, Table 2). As noted in the abstract, autism was also correlated with residential proximity to a freeway during the third trimester (OR, 2.22, CI, 1.16-4.42). Note that the confidence interval almost encompasses 1.0. For purposes of this study, freeways were defined as U.S. interstate highways or state highways. You might also wonder why 309 m was chosen as the distance cutoff. I wondered that too when I first read news accounts of this study. My first thought was that 309 m was equivalent to 1,000 ft., and so it is–approximately. However, it’s not close enough to explain why 309 m was chosen as the cutoff, given that 309 m equals 1,013.8 ft., and 1,000 ft. equals 304.8 m. On the surface, it looked like a bit of the old data dredging to find a distance that gives a “significant” result. However, this is how the manuscript described the rationale for choosing the distances:
We examined the distribution of distance from the nearest freeway among subjects in our study and determined exposure cut-points to define the closest 10% (<309m), subsequent 15% (309m-647m), and following 25% (647m-1419m) as exposure groups. The remaining 50% (>1419m) served as the reference group in our analysis. Living within 309m of a freeway at birth was associated with autism (odds ratio (OR) 1.86, 95% confidence interval (CI) 1.04-3.45). This association was not altered by adjustment for child gender or ethnicity, maximum education in the home, maternal age, or maternal smoking during pregnancy (Table 2). When we categorized our distance measure into deciles, only the top 10% corresponding to the <309m category above, showed evidence of an increased autism risk compared to those living farthest from the freeway (lowest decile, >5,150m) (unadjusted OR=2.48, 95%CI 1.17-5.39).
Now, here’s the anomaly. This very same study failed to find any correlation with living near what is defined as a “major road,” specifically a state highway, interstate highway, or major arterial. This was true even though the distances from a major road of the closest 10%, 25%, and 50% in this study were 42 m, 96 m, and 209 m, respectively, all much closer. If pollution is the common factor that accounts for the observation of a relative risk for being diagnosed with autism of 2-fold greater than in children farther away from freeways, then these results are meaningless without knowing the levels of pollution at 42 m from a major road compared to 309 m from a freeway. The authors try to explain this anomaly away by doing some hand waving about the higher traffic volumes carried by Southern California freeways, which might well be true, but proximity matters. If proximity to a freeway is a surrogate for exposure to pollution (or a certain subset of pollutants), is it so unreasonable to expect that being more than seven times closer to a major road compared to a freeway would result in exposure to as much pollution? I don’t know the answer to this, although this sort of data is known. For instance, it’s known that resuspension by Los Angeles Interstate 405 significantly increases atmospheric particle numbers and metal concentrations but that both return to urban background levels through dry deposition within 100–150 m of the highway (Sabin et al., Atmos Environ 2006;40:7528–7538).
One thing’s for sure, though. The fact that the investigators didn’t see even a hint of a correlation between autism and a similar measure to proximity to freeways makes me wonder very strongly whether this is simply nothing more than a finding that popped up in multiple comparisons due to random chance alone.
Let’s, for argument’s sake, however, assume that this isn’t the case. Let’s assume that the data are the data, and the correlation found appears not to be due random chance alone, at least using the data and techniques used in this study. I stil see a big problem: a confounding factor that wasn’t controlled. for. True, the authors did try to control for all the usual confounders in autism studies: child sex, ethnicity, level of education in the home, maternal smoking, maternal age, and preterm delivery. (One notes that they didn’t control for paternal age, which, as recent studies have shown, appears to correlate with an increased risk of autism.) But what did they fail to control for? If you remember my deconstruction of Dr. Raymond F. Palmer’s study that claimed to link proximity to mercury-emitting power plants to an increased risk for autism. Do you remember what that was? No reason to expect that you should, given that my post is over two and a half years old, but the criticism of Palmer’s study applies equally well here. Moreover, Joseph explicitly listed a number of complaints about Palmer’s study and demonstrated that the correlation that Palmer found disappeared once population density was accounted for. Why is this important? Urban and suburban regions with much denser populations tend to have more screening and intervention programs for autism and thus a higher rate of diagnosis. If we take a look at the distribution of CHARGE study subjects from the Sacramento Valley catchment area, it can be seen that they are scattered throughout central California. I couldn’t find an equivalent map for the Los Angeles basin catchment area.
The reason why controlling for population density and urbanicity is so important was well explained by Thomas A. Lewandowski in criticizing the Palmer study:
Lastly, the authors found that the most important determining factor for autism prevalence in their study was whether the child lived in an urban, suburban, or rural area. For example, residence in an urban school district resulted in a 473% higher rate of autism compared to rural districts. Similar findings have been reported by others (e.g., Deb and Prasad, 1994). The urbanization effect is nearly 8 times stronger than the effect suggested for mercury but is given relatively little discussion and is not even noted in the abstract. Since levels of many pollutants (including mercury) would be strongly correlated with urbanization/industrialization, this also leads one to question the mercury-autism association the authors report…Certainly a host of environmental and social variables associated with urbanization could be investigated as possible factors in autism. Alternatively, an increased tendency for diagnosis in urban localities could explain at least part of the increased incidence.
Urbanicity is also associated with proximity to Interstate freeways, although, to be fair, it is also associated with proximity to major roads. The denser the population, the more freeways and roads.
Another question that came up as I read this study was a simple one: How many total associations were the authors looking for in their data? Remember, the CHARGE study is looking for environmental factors that modulate autism risk, as described in this review article about the study published in 2006, coauthored by one of the study under discussion:
To structure the search for etiologic factors, we are beginning with known neurodevelopmental toxicants and hints from the immunologic evidence. Additionally, physiologic differences that might provide clues about susceptibility and mechanisms are being examined through characterization of metabolic, immunologic, and gene expression profiles, as well as genetic polymorphisms. Figure 1 shows five broad classes of exposures of interest: pesticides, metals, persistent pollutants with known or suspected neurodevelopmental or immunologic toxicity, medications and other treatments, and infections. Exposures from both the prenatal and early childhood periods are being investigated, with data primarily from three sources: a) extensive interviews with parents; b) laboratory analysis of xenobiotics in blood, urine, and hair specimens; and c) prenatal, labor and delivery, neonatal, and pediatric medical records.
This is represented in a figure:
In other words, the current paper is a report of a subset of analyses from a much bigger study.
The bottom line
Given that this published study is part of a much larger study looking at a lot of factors, there’s always the danger that the reported finding is nothing more than an anomalous result that can be attributed to random chance alone. Such findings are quite common when multiple comparisons are made; indeed, as the number of comparisons rises, so does the chance of finding a fluke correlation that mathematically is statistically significant but is in reality merely random chance in action. In other words, if investigators were looking for 100 different associations and found this one, it would be a different thing than if they were looking for only five and found this one. Without knowing the number of associations authors were looking for at the beginning without having to go to various other papers describing the study, it’s hard to judge whether this one correlation (autism prevalence with proximity to freeways either at birth or during the third trimester, which, let’s face it, correlate closely with each other, given that most expectant mothers don’t want to change residences when they are near the end of their pregnancy) means anything at all. My guess is that it probably does not; I would find the data more convincing if there were included actual measurements of pollutants compared to proximity to freeways and major roads, along with an analysis that shows that pollutants up to 309 m away from a freeway are still much higher than pollutants a mere 42 m from a major road. It would also have been a lot more convincing if there were a geographic effect consistent with the direction that the prevailing winds blow.
Epidemiological studies of clusters of conditions or diseases are fraught with false positives, so much so that there is even a name for the particular fallacy that all too many investigators fall prey to when looking for such clusters: The Texas sharpshooter fallacy. The fallacy is named for the apocryphal Texas sharpshooter who fired a bunch of bullets into the side of a barn and then drew a circle around the area where most of the bullets hit, calling that circle the bullseye. Similarly, epidemiologists look at the geographical location of subjects with a particular condition or disease, and then try to “draw a circle” around them that fits in with what they are looking for. If that’s what happened here, then the CHARGE investigators, believing that pollution might be a contributor to autism, tried to “draw the circle” near freeways and major roads, found one statistically significant result, and then concluded that proximity to freeways is a risk factor for autism, the implication of which (to the investigators) is that pollution is a predisposing factor to autism. I can’t help but note in looking at these results that it is is already known from a study earlier this year that in California autism cases tend to cluster in areas with high populations of more affluent, educated people, who, not surprisingly, tend to live in more affluent communities with good school services, special education, and autism screening and intervention programs.
This approach of examining geographic clusters is nearly always fallacious because (1) the cluster is usually the result of random chance or (2) there may be other reasons for the clustering. Personally, I suspect option #2, with the other reason for the clustering being the unaccounted-for confounding factor of urbanicity. In all fairness, the authors do mention in the discussion that their result might be due to “random chance or bias.” Also, in various interviews, Volk did say that their work needs to be replicated (although not, as far as I saw, that their results are likely to be due to random chance or an uncontrolled-for confounding factor). However, missing from most news reports was a level of skepticism about a study finding in essence only one barely statistically significant result among several.
One can’t help conclude that, if this association were true, then, as AutismNewsBeat pointed out, we would expect autism prevalence to have started its upward trend back in the late 1950s, after the passage of the Interstate Highway Act and the subsequent explosion of construction of interstate freeways. As we all know, however, this explosion didn’t actually begin until the early 1990s, which is the reason why anti-vaccine groups seized upon vaccines as the One True Cause of autism. Such are the hazards of confusing correlation with causation. I don’t think that that’s what the authors are necessarily doing here, but I do think they are doing a disservice to the public by not adequately trying to squash the misunderstandings about studies of epidemiological clusters.

   Mathematically modeling why quackery persists
David Gorski
Link
It’s often puzzled me (and, I daresay, many other skeptics and boosters of science- and evidence-based medicine) why various forms of quackery and woo that have either about as close to zero prior probability as you can get and/or have failed to show evidence greater than placebo in clinical trials manage to retain so much traction among the public. Think homeopathy. Think reiki. The former is nothing more than sympathetic magic prettied up with science-y sounding terms, while the latter is nothing more than faith healing given a slant based on Eastern mysticism and religion instead of Christianity. Indeed, reiki was even inspired by stories of Jesus’ healing powers, complete with a trip into the wilderness for fasting and prayer, resulting in revelation. Or consider acupuncture, a modality that is seemingly more popular than ever, even invading the very sanctum sanctorum of the ivory towers of academic medicine, yet every study of which that is done under rigorous conditions with proper placebo controls shows it to be no more efficacious than a placebo. It’s easy enough to shake one’s head and chalk it up to irrationality, ignorance of science, or even religious faith, but I’ve always been dissatisfied with such glib explanations, even though admittedly I have myself used them on occasion.
That’s why a study released last week in PLoS One by Mark M. Tanaka, Jeremy R. Kendal, Kevin N. Laland out of the Evolution & Ecology Research Centre, School of Biotechnology & Biomolecular Sciences, University of New South Wales, the Department of Anthropology, University of Durham, and the School of Biology, University of St Andrews, Fife, respectively, entitled From Traditional Medicine to Witchcraft: Why Medical Treatments Are Not Always Efficacious. Besides loving the title, I also like the methodology, which in essence adapts the tools of modeling evolution and the spread of traits throughout a population and asks the question: Why do ineffective or even harmful (or, as the authors characterize them, “maladaptive”) treatments for various illnesses persist in populations? The results are surprising and counterintuitive, yet ring true. In essence, the authors conclude that the most efficacious self-treatments are not always the ones that spread and that even harmful treatments can spread. Both of these observations are entirely plausible based on the prevalence of usage of common woo and quackery, and what the authors have done, in essence, is to model mathematically why quackery persists.
Indeed, the authors set the stage:
In recent years, 60–80% of the world’s population, mainly from developing countries, depended primarily on traditional medicines, folk remedies and home cures, as well as treatment from witchdoctors and other ‘supernatural practices’, for their health-care needs [1]. In western societies, complementary and alternative medicine is garnering increasing interest and acceptance. At current growth rates, two-thirds of Americans are projected to be using alternative medicine by 2010 [2]. Asian governments are pouring billions of dollars into screening Traditional Chinese medicines in the hope that clinical trials will spawn lucrative drugs [3]. Traditional medicine has become big business.
Indeed, it has, which is why I always laugh when advocates of “complementary and alternative medicine” (CAM) claim it’s all about the money for big pharma but not for CAM. But I digress:
While scientific studies have validated some traditional remedies, for instance, by confirming the biological activity of plant extracts [4], [5], the use of complementary and traditional medicines remains contentious, and doubts about the efficacy and safety of many treatments remain [1], [6], [7], [8]. Reservations over safety and efficacy underpin controversy over USA and UK universities’ attempts to bring alternative medicines into medical school curricula [9]. The active ingredients used in many traditional medicines are potentially toxic, often containing dangerous elements, including heavy metals [5], [10]. Even the use of ineffective non-toxic remedies can be harmful if it delays effective treatment. For instance, fears have been expressed that, in Nigeria, witchcraft and traditional remedies of unknown efficacy are widely employed as treatments for malaria, instead of, or delaying access to, modern medicines of proven effectiveness [11]. In sub-Sarahan Africa there is a concern that the use of traditional remedies for mastitis, a condition often attributed to sorcery, may inadvertently be contributing to the spread of HIV [12].
In 2002 the WHO [1] launched a global plan to make the use of traditional medicine safer by encouraging evidence-based research on the safety, efficacy and quality of traditional practices. Accordingly, traditional medicines are currently undergoing scrutiny to evaluate their effectiveness and monitor adverse events [3], [13]. Such analyses have often failed to confirm the efficacy of traditional remedies: for instance, of nearly 25,000 applications for registration of traditional medicines received by Malaysian authorities, 37.3% were rejected, either on grounds of safety or ineffectiveness [14]. However, there is currently no compelling explanation for the prevalence of low-efficacy treatments.
As I said before, I never found explanations of lack of scientific understanding or confusing correlation with causation entirely satisfactory. Don’t get me wrong; I definitely think these are major reasons why, for instance, so many parents believe vaccines cause autism or so many people think that reiki works. I just don’t think it’s enough.
Enter Tanaka et al. They basically construct a model in which they attempt to estimate the factors that could affect the spread of various treatments throughout the population. One aspect of this model to be kept in mind is that it primarily looks at self-medication, although parents medicating their children, as happens in too much autism quackery, could be included under this category, mainly because it is not the children who decide for themselves whether a treatment is working or not; it is the parents. The general assumptions of the model are summarized in Figure 1 below:
Table 1 lists the parameters inputted into the model:
I’m not going to go into the equations in much detail. In fact, I’m not going to go into the equations at all. The article is freely available to all; so you can read it yourself. What I will discuss are some of the assumptions going into the model. First among these is this:
We assume that individuals are either in a diseased state or in a healthy state. We model the spread of a behavioural trait expressed in treatment of disease. The behavioural trait in question is any innovation, practice or treatment that could potentially affect the outcome of this disease. To model the spread of a behavioural trait, we make the following assumptions. A new behavioural trait arises in (or is invented by) an ill individual who may then demonstrate this practice; others who are ill may adopt the practice upon being exposed to it, and then become demonstrators themselves. In other words, demonstrators convert observers. There is empirical support for the assumption that self-medicative treatments spread through social learning [22].
The most important assumption made for the model is that observers adopt the trait in an unbiased fashion. The authors don’t assume a prior that efficacy has anything to do with whether observers take up a treatment. They justify this by using the quite sensible reasoning that observers are not very good at judging the efficacy of a treatment. Any regular reader of this blog should be aware of why this is true; it’s basically because any observer trying to determine whether a treatment is efficacious on the basis of watching it being used, is using anecdotal evidence to come to a conclusion. If there’s one theme that’s run through this blog from the very beginning, it’s how easily we as humans are deceived by anecdotal evidence. Indeed, unless a remedy is rapidly fatal or produces a miraculous recovery, given the effects of regression to the mean, confirmation bias, treatment bias, and a variety of other biases, judging the efficacy of a therapy on the level of an individual can be very problematic even for physicians. For the untrained, it’s really tough.
The authors define the “cultural fitness” of a treatment as the mean number of converts produced by a demonstrator and then studied what the value of cultural fitness would be under various assumptions, both for instances of single illness and in the case of multiple relapses of illness. Interestingly, although highly efficacious treatments are predicted to spread through the population, even in the case of inefficacious treatments, multiple exposures of it and a remedy for it actually increase the chance that the treatment will spread, reagardless of whether the treatment is efficacious or not. The authors concluded that, although highly efficacious treatments have a high cultural fitness, treatments with efficacy near or at zero can also spread. The conditions under which such remedies could spread and achieve high cultural fitness are when treatments are primarily demonstrated in sickness and demonstrate low abandonment, particularly where the likelihood of relapse is small. The authors summarize:
This study offers a simple, novel and counter-intuitive hypothesis for the prevalence of ineffective medical treatments: unbiased copying of new treatments can frequently lead to the prevalence of ineffective practices because such treatments are demonstrated more persistently than efficacious alternatives, even when there is enhanced abandonment of ineffective cures. By unbiased copying, we mean copying in direct proportion to the rate at which the alternative variants are demonstrated. Here, in simple terms, treatment frequency dynamics are typically dominated by two processes, representing the rates of acquisition and loss of remedies. Maladaptive and superstitious treatments can become prevalent because their ineffectiveness prolongs illness, enhancing their rate of demonstration relative to efficacious cures, and leading to elevated rates of acquisition that may compensate for greater loss.
Our finding that superstitious treatments can easily spread is supported by reports of extraordinary treatments for conditions such as leprosy (treated with a drink made of rotting snakes) and syphilis (treated by eating a vulture), and by similar myths for poisonous snake bites (apply ‘guaco’ leaves, poisonous lizard skin or snake’s bile), dog bites (drink tea made from the dog’s tail) and scorpion stings (tie a scorpion against the stung finger) [23]. The analysis also helps explain the persistence of medical treatments of animals, such as ‘firing’ (cautery) of working horses, employed for millennia as treatment for lameness, where recovery is rare, and still widely practiced in many countries in spite of trials establishing its ineffectiveness [24].
Another conclusion of this model is that even highly effective treatments can be lost due to stochasticity, in other words, due to random chance. Indeed, the authors point out that most highly efficacious innovations would be predicted to be lost due to stochasticity. Indeed, my reading of this study would suggest that one reason why highly effective treatments actually do persist in our society is because scientists, physicians, and science-based medicine validate what treatments are efficacious, retaining the treatments that are and trying (but not always succeeding) in discarding the ones that aren’t. For people self-medicating, such a mechanism is not operative.
One aspect of this model that I consider inadequate is that it doesn’t take into account the role of peer pressure and groupthink. It certainly doesn’t take into account the rapidity with which glowing stories of success for quackery can spread through both word of mouth and other methods. In this case, the Internet truly has changed everything. Never before, has it been possible for so many people to “listen to so many stories” in such a short period of time or to form online communities made up of people who mutually reinforce each other’s beliefs, even though most of them have never met face-to-face.
I return to one of my favorite examples, autism quackery such as “biomedical interventions” and chelation therapy. Because autism is a condition of developmental delay, not stasis, it’s very easy to confuse correlation with causation and believe that a given ineffective intervention “worked.” Prometheus points out this very phenomenon:
If you have the opportunity, check in to one of the many “biomedically oriented” Internet groups and see what happens when somebody questions the idea that “biomed” can “recover” autistic children. At the least, they will be admonished – “Don’t stand in the way of other parents getting their children the help they need!”. More likely, they will be told to “Shut up!” and banned from the group. In some cases, they will be harassed and even threatened.
I’m not sure how the authors could have modeled the “Emperor’s New Clothes” phenomenon, such as Prometheus describes, but I can’t help but think that such mechanisms are at work in addition to the ones described in this paper.
As much as I like this paper for its description of a plausible mechanism by which implausible and ineffective treatments can spread through a population and remain entrenched, I have to caution that, without some sort of experimental or observational validation, it remains just that: a model. Unfortunately, I’m having some difficulty figuring out just how one might test this model against reality in a real human population.
One thing that this model leads me to wonder, though, is whether, maybe–just maybe–there is a force that can prevent or halt the spread of implausible, unscientific, and ineffective treatments, such as homeopathy, throughout the population. Perhaps such a method both serves as a memory that recalls which therapies don’t work and a method of testing which methods do work. That method is science-based medicine.
I wonder how the effect of SBM can be incorporated into this model.
REFERENCES:
Tanaka, M., Kendal, J., & Laland, K. (2009). From Traditional Medicine to Witchcraft: Why Medical Treatments Are Not Always Efficacious PLoS ONE, 4 (4) DOI: 10.1371/journal.pone.0005192

   Skepticism versus nihilism about cancer and science-based medicine
David Gorski
Link
Last Friday, Mark Crislip posted an excellent deconstruction of a very disappointing article that appeared in the most recent issue of Skeptical Inquirer, the flagship publication of the Committee for Skeptical Inquiry (CSI). I say “disappointing,” because I was disappointed to see SI (Skeptical Inquirer, not Sports Illustrated) publish such a biased, poorly thought out article, apparently for the sake of controversy. I’m a subscriber myself, and in general enjoy reading the magazine, although of late I must admit that I don’t always read each issue cover to cover the way I used to do. Between work, grant writing, blogging, and other activities, my outside reading, even of publications I like, has declined. Perhaps SI will soon find itself off my reading list. Be that as it may, I couldn’t miss the article that so irritated Mark, because it irritated me as well. There it was, emblazoned prominently on the cover of the March/April 2011 issue: Seven Deadly Medical Hypotheses. I flipped through the issue to the article to find out that this little gem was written by someone named Reynold Spector, MD. A tinge of familiarity going through my brain, I tried to think where I had heard that name before.
And then I remembered.
Dr. Spector, it turns out, first got on my nerves about a year ago, when he wrote an article for the January/February 2010 issue of SI entitled The War on Cancer: A Progress Report for Skeptics. I remember at that time being irritated by the article and wanting to pen a discussion of the points raised but don’t recall why I never actually did. It was probably a combination of the fact that SI doesn’t publish its articles online until some months have passed after the paper version has been released and perhaps my laziness about having to manually transcribe with my own fingers any passages of text that I might want to cite. By the time the article was available online, I forgot about it and never came back to it–until now. I should therefore, right here, right now, publicly thank Mark (and, of course, Dr. Spector) for providing me the opportunity to revisit that article in the context of piling on, so to speak, Dr. Spector’s most recent article. After all, Deadly Hypothesis Seven (as Dr. Spector so cheesily put it) is:
From a cancer patient population and public health perspective, cancer chemotherapy (chemo) has been a major medical advance.
Dr. Spector then takes this opportunity to cite copiously from his 2010 article, sprinkling “(Spector, 2010)” throughout the text like powdered sugar on a cupcake. There’s the opening I needed to justify revisiting an article that’s more than a year old! And what fantastic timing, too, hot on the heals of my post from a couple of weeks ago entitled Why haven’t we cured cancer yet?
Back to the future, before visiting the past
Before I leap back more than a year, first let me just point out that I agree with Mark 100% when he castigated Dr. Spector for his opening volley. Although thought had never occurred to me at the time, in retrospect I even agree with Crislip’s likening Dr. Spector’s tirade to a Mike Adams screed, at least in dogmatic certainty and tone. Sadly, for entertainment value at least, Dr. Spector lacks the looney hyperbole of Mike Adams. Also, unlike Adams, Dr. Spector is a real scientist, but damn if, references omitted, this first sentence couldn’t have come right out of a Mike Adams special:
A chronic scandal plagues the medical and nutritional literature (Spector and Vessell, 2006; Spector, 2009): much of what is published is erroneous, pseudoscientific, or worse.
Like Mark, I’m a John Ioannidis convert, and I accept that there is a lot of medical literature that is erroneous. (Just search for Dr. Ioannidis’ last name on this blog, and you’ll find copious posts praising him and discussing his work.) In fact, as I’ve pointed out, most medical researchers instinctively know that most new scientific findings will not hold up to scrutiny, which is why we rarely accept the results of a single study, except in unusual circumstances, as being enough to change practice. I also have pointed out many times that this is not necessarily a bad thing. Replication is key to verification of scientific findings, and more often than not provocative scientific findings are not replicated. Does that mean they shouldn’t be published?
As for pseudoscience, I’m half tempted to agree with Dr. Spector, but just not in the way he thinks. Unfortunately, over the last 20 years or so, there has been an increasing amount of pseudoscience in the medical literature in the form of “complementary and alternative medicine” (CAM) studies of highly improbable remedies or even virtually impossible ones (i.e., homeopathy). However, that does not appear to be what Dr. Spector is talking about, which is why I looked up his references. The second reference is to an SI article from 2009 entitled Science and Pseudoscience in Adult Nutrition Research and Practice. There, and only there, did I find out just what it is that Dr. Spector apparently means by “pseudoscience”:
By pseudoscience, I mean the use of inappropriate methods that frequently yield wrong or misleading answers for the type of question asked. In nutrition research, such methods also often misuse statistical evaluations.
Ah, now I get it! Dr. Spector doesn’t really know the difference between inadequately rigorous science and pseudoscience! Now, don’t get me wrong. I know that it’s not always easy to distinguish science from pseudoscience, especially at the fringes, but in general bad science has to go a lot further than Dr. Spector thinks to merit the the term “pseudoscience.” It is clear (to me, at least) from his articles that Dr. Spector throws around the term “pseudoscience” around rather more loosely than he should, using it as a pejorative for any clinical science less rigorous than a randomized, double-blind, placebo-controlled trial that meets FDA standards for approval of a drug (his pharma background coming to the fore, no doubt). Pseudoscience, Dr. Spector. You keep using that word. I do not think it means what you think it means. Indeed, I almost get the impression from his articles that Dr. Spector views any study that doesn’t reach FDA-level standards for drug approval to be pseudoscience. He also piles on nutritional science (that is, after all, his area of specialty), and there is no doubt that here is a lot of pseudoscience there; however the vast majority of it is not in the scientific literature. Rather, it’s snake oil salesmen selling diets and supplements based on less rigorous science, the willful twisting and misinterpretation of acceptable science, and making up claims out of whole cloth.
Here’s the deal. Medical science, when it works well, tends to progress from basic science, to small pilot studies, to larger randomized studies, and then–only then–to those big, rigorous, insanely expensive randomized, double-blind, placebo-controlled trials. Dr. Spector mentions hierarchies of evidence, but he seems to fall into a false dichotomy, namely that if it’s not Level I evidence, it’s crap. The problem is, as Mark pointed out, in medicine we often don’t have Level I evidence for many questions. Indeed, for some questions, we will never have Level I evidence. Clinical medicine involves making decisions in the midst of uncertainty, sometimes extreme uncertainty. One example is hormone replacement therapy for menopause before the results of the Women’s Health Initiative study were reported. Indeed, Dr. Spector makes much hay out of the widespread use of hormone replacement therapy (HRT) back in the 1980s and 1990s, as an example of what happens when we don’t adhere to proper clinical trial design, labeling Deadly Hypothesis Two as:
The hypothesis was that if women replace these hormones postmenopausally with HRT, they will remain “youthful” and not suffer from heart disease, dementia, vaginal dryness, hot flashes, and fractured bones.
Dr. Spector then proceeds to paint a picture of reckless physicians proceeding on crappy studies to pump women full of hormones. Actually, it was more than a bit more complicated on than that. That was the time when I was in my medical training, and I remember the discussions we had regarding the strength (or lack thereof) of the epidemiological data and the lack of good RCTs looking at HRT. I also remember that nothing works as well to relieve menopausal symptoms as HRT, an observation we have been reminded of again since 2003, which is the year when the first big study came out implicating HRT in increasing the risk of breast cancer (more later). It’s also hard not to point out that, these days, it’s not physicians promoting HRT. Rather, it’s women like Suzanne Somers promoting “bioidentical hormones” and telling postmenopausal women that they should strive for the estrogen levels that they had when they were 25. Be that as it may, I found a rather fascinating editorial in the New England Journal of Medicine from more than 20 years ago that discussed the state of the evidence back then with regard to estrogen and breast cancer:
Evidence that estrogen increases the risk of breast cancer has been surprisingly difficult to obtain. Clinical and epidemiologic studies and studies in animals strongly suggest that endogenous estrogen plays a part in causing breast cancer. If so, exogenous estrogen should be a potent promoter of breast cancer. Although more than 20 case–control and prospective studies of the relation of breast cancer and noncontraceptive estrogen use have failed to demonstrate the expected association, relatively few women in these studies used estrogen for extended periods. Studies of the use of diethylstilbestrol and oral contraceptives suggest that a long exposure or latency may be necessary to show any association between hormone use and breast cancer. In the Swedish study, only six years of follow-up was needed to demonstrate an increased risk of breast cancer with the postmenopausal use of estradiol. It should be noted, however, that half the women in the subgroup that provided detailed data on the duration of hormone use had taken estrogen for many years before their base-line prescription status was defined. The duration of estrogen exposure in these women before the diagnosis of breast cancer was probably seriously underestimated; a short latency cannot be attributed to estradiol on the basis of these data. Other recent studies of the use of noncontraceptive estrogen suggest a slightly increased risk of breast cancer after 15 to 20 years’ use.
One notes that, even now, the evidence is conflicting regarding HRT and breast cancer, with the preponderance of evidence suggesting that mixed HRT (estrogen and progestin) significantly increases the risk of breast cancer, while estrogen-alone HRT very well might not increase the risk of breast cancer at all or (more likely) only very little. Indeed, I was just at a conference all day Saturday where data demonstrating this very point were discussed by one of the speakers. None of this stops Dr. Spector from categorically labeling estrogen as a “carcinogen that causes breast cancers that kill women.” Maybe. Maybe not. It’s actually not that clear. The problem, of course, is that, consistent with the first primary reports of WHI results, the preponderance of evidence finding health risks due to HRT have indicted the combined progestin/estrogen combinations as unsafe. Even at this late time, we do not truly know whether estrogen-alone HRT carries significant risks (there are even some studies, a subgroup of the WHI included, that hint at a slightly protective effect of estrogen against breast cancer), but we do have a pretty good idea that estrogen-alone HRT is almost certainly safer than estrogen-progestin HRT, at least with respect to breast cancer. Unfortunately, there is good evidence that estrogen-only HRT increases the risk of uterine cancer. Either way, Dr. Spector’s dogmatic and unnuanced discussion of the issue suggests an axe to grind more than an interest in providing an accurate picture. Indeed, he even goes so far as to slip this sentence in there: “In legal challenges the U.S. Supreme Court has upheld the notion that HRT causes breast cancer.”
What? Who cares what the Supreme Court says about medical causation? That’s a legal standard, not a scientific standard! There’s no reason to throw such a sentence into a serious medical article.
The rest of the “Seven Deadly Medical Hypotheses” remain just as questionable, particularly the way Dr. Spector castigates the idea that screening populations for breast and prostate cancer saves lives. In all fairness, Dr. Spector’s skepticism towards screening is actually the least unreasonable part of his most recent article, given recent evidence. He even concludes that mammography has a small benefit. Therein lies a contradiction. If mammography, for instance, is beneficial, even if only slightly, then how can it be part of a “deadly medical hypothesis”? Calling screening a “deadly” medical hypothesis is, as is the case with much of this article, way over the top. A “disappointing” medical hypothesis would be a far better description. In any case, for a far more nuanced discussion of cancer screening, I will blow my own horn and suggest that you go to read these posts:
Posts by yours truly:
The early detection of cancer and improved survival: More complicated than most people think
Early detection of cancer, part 2: Breast cancer and MRI
Do over one in five breast cancers detected by mammography alone really spontaneously regress?
Are one in three breast cancers really overdiagnosed and overtreated?
The cancer screening kerfuffle erupts again: “Rethinking” screening for breast and prostate cancer
The USPSTF recommendations for breast cancer screening: Not the final word
The mammography wars heat up again
Molecular breast imaging (MBI): A promising technology oversold in a TED Talk?
Posts by Harriet Hall:
Screening Tests – Cumulative Incidence of False Positives
Overdiagnosis
By Val Jones:
The Mammogram Post-Mortem
Read them now, or read them later. They are far more informative and nuanced than anything Dr. Spector has written on these topics for SI. For example, you won’t find the word “deadly” in there anywhere. In the meantime, let’s move on to the article that irritated me so over a year ago. Oddly enough, it was a bit more balanced and less dogmatic than the current article.
“We’re losing the war on cancer”?
The beginning of The War on Cancer A Progress Report for Skeptics, while not as over-the-top as the beginning of Seven Deadly Medical Hypotheses, is still pretty negative:
In 1971, President Nixon and Congress declared war on cancer. Since then, the federal government has spent well over $105 billion on the effort (Kolata 2009b). What have we gained from that huge investment? David Nathan, a well-known professor and administrator, maintains in his book The Cancer Treatment Revolution (2007) that we have made substantial progress. However, he greatly overestimates the potential of the newer so-called “smart drugs.” Re searchers Psyrri and De Vita (2008) also claim important progress. However, they cherry-pick the cancers with which there has been some progress and do not discuss the failures. Moreover, they only discuss the last decade rather than a more balanced view of 1950 or 1975 to the present.
On the other hand, Gina Kolata pointed out in The New York Times that the cancer death rate, adjusted for the size and age of the population, has decreased by only 5 percent since 1950 (Kolata 2009a). She argues that there has been very little overall progress in the war on cancer.
I find it rather amusing how Dr. Spector accuses investigators like Dr. Vincent DeVita (who is a coauthor of one of the widely used textbooks of oncology there is, a copy of which sits on my shelf in my office) of “cherry picking” cancers for for which progress has been made and focusing primarily on the last decade while at the same time stating that he is going to focus on adult solid cancers, where less progress has been made. One might argue that this is a correct way to do it, and far be it from me to claim that the war on cancer has been an overwhelming success. As I pointed out a couple of weeks ago, however, there is a reason (many reasons, actually) why progress has been so elusive. There is also at least one other reason. However, it’s quite obvious in both articles that Dr. Spector has himself concentrated on data that he can use to make the situation look as dismal as possible; i.e., to support his apparently preconceived conclusions that the war on cancer has been a miserable failure and that chemotherapy doesn’t work (that is, except when it does work very well). Indeed, in his most recent article, Dr. Spector uses terminology that is even more negative about chemotherapy than in his previous article. For example in the current article, Dr. Spector writes:
The hypothesis behind current chemotherapy is deeply flawed. How can we expect these drugs, most of which are nonspecific cellular poisons, to kill only wiley cancer cells and not normal cells? In fact, cacner chemotherapy routinely kills bone marro and the cell lining of the gastrointestinal tract, with very distressing (indeed sometimes fatal) side effects.
It rather shocks me that a pharmacologist like Dr. Spector would write something like this without mentioning that the reason chemotherapy can work in some tumors is because it is more toxic to the cancer cell than it is to (most of) the surrounding normal tissue. The same thing is true of radiation. This is not a trivial point. Dr. Spector makes it sound as though chemotherapy poisons everything equally and that it’s a miracle that it ever works at all. He then writes:
However, it cannot be denied that there are a few populations for which chemotherapy is marvelously effective, as noted above, and must be used.
No kidding! For example, many leukemias and lymphomas, testicular cancer, and anal cancer (which is treated primarily with the Nigro protocol, which consists of chemotherapy and radiation), among others!
Dr. Spector correctly points out that smoking tobacco products is a major cause of lung cancer, which causes most cancer deaths these days in both men and women. He even shows several very nice graphs that demonstrate that lung cancer death rates began to skyrocket about twenty years after cigarette consumption began to skyrocket (Figure 3) and that lung cancer death rates have been declining (in men at least; they appear to be peaking in women, Figures 4 and 5), approximately 25 years after cigarette consumption began to decline. All of these results are very much in accord with what we know about cigarette smoke as a carcinogen. One thing is clear: If we could eliminate cigarette smoking completely, approximately 20 years later, declines in lung cancer deaths would drive major declines in overall cancer deaths. There’s little doubt of that. However, such an observation is very much a “Well, duh!” conclusion. It’s not as though public health authorities in the U.S are unaware of such statistics or that dramatic gains or haven’t been working to try to discourage smoking. Changing behavior is very, very hard. In fact, I would argue that decreasing cancer mortality by 5% since 1950 is actually pretty darned good, given that lung cancer mortality tripled among men between 1950 and 1990 and more than doubled among women from 1970 to 2000. If you don’t believe me, take a look at these graphs from the SEER Database of cancer death rates from the various major forms of cancer, in particular Figure 4 and Figure 5. Then there is this graph in Figure 2:
This demonstrates that, although cancer incidence rates are leveling off, cancer death rates are decreasing. Sure, it’s not as good as we need to do, but notice that the curves are separating. Indeed, if you look at Figure 7, you’ll see an estimate of how high cancer mortality would be if cancer mortality rates had remained the same over the last 20 years:
The American Cancer Society attributes the decreases in cancer mortality to this:
The decrease in lung cancer death rates among men is due to a reduction in tobacco use over the past 50 years, whereas the decrease in death rates for female breast, colorectal, and prostate cancer largely reflects improvements in early detection and/or treatment. Between 1990–1991 and 2006, death rates increased for liver cancer in both men and women, esophageal cancer and melanoma in men, and lung and pancreatic cancer in women. Figure 7 shows the total number of cancer deaths avoided since death rates began to decrease in 1991 in men and in 1992 in women. Approximately 767,000 cancer deaths (561,400 in men and 205,700 in women) were averted between 1991–1992 and 2006.
Death rates from all cancer are a rather crude measure of how the war on cancer is going, anyway, being an aggregate number that is only likely to be affected very late as a result of improvements in cancer therapy. Our population is aging, which means that more people are living long enough to develop cancer, particularly given that we’ve made such dramatic progress in decreasing death rates from cardiovascular disease. People who live longer because they don’t die of cardiovascular disease have to die of something, and cancer, being number two (although not for long), is the next most likely cause. In any case, let me provide an example of what I find wrong with Dr. Spector’s thesis. In his “war on cancer” article a year ago, Dr. Spector wrote:
The FDA has approved bevacizumab for the cancers listed in table 5 (Physicians Desk Reference [PDR] 2009; Health Agencies Update 2009). Since the median survival of colorectal cancer is eighteen months, bevacizumab therapy would cost about $144,000 (in such a patient) for four months prolongation of survival (Keim 2008). In the other cancers in table 4, there is no prolongation of survival. Moreover, bevacizumab can have terrible side effects, including gastrointestinal perforations, serious bleeding, severe hypertension, clot formation, and delayed wound healing (PDR 2009). By the criteria in table 4, bevacizumab is at best a marginal drug. It only slightly prolongs life, demonstrable only in colorectal cancer, has serious side effects, and is very expensive.
Here’s a more provocative and somewhat more optimistic assessment of the progress we’ve made in treating metastatic colorectal cancer in graph form (click to make bigger):
Untreated metastatic colorectal cancer has about a four to six month median survival. Our old chemotherapy regimens increased the medical survival in metastatic colorectal cancer to around 12 months. Newer regimens boosted that to around 16 or 17 months. Adding bevacizumab then increased the survival to around 21 months. All of this has occurred since the mid-1990s. Is it enough? Of course not! But as a clinician I would argue that quadrupling the life expectance of someone with metastatic colorectal cancer is nothing to sneeze at. Indeed, I even have personal experience with this. The father of a good friend died last summer of metastatic colorectal cancer after having lasted over six years. A combination of surgeries, chemotherapy, and other treatments kept him going. He played golf until a week before his health declined precipitously and unexpectedly, and he died. Yes, he was certainly an outlier, but if the median survival of a group of cancer patients is pushed longer and longer, there will be more and more outliers like my friend’s father. True, we would much prefer a cure (or at least a reasonable chance of a cure), but ask the cancer patient what that extra year or several months means to him or her. Also true, as a society, we have to decide how much we are willing to pay for that additional survival, but that is a societal question more than anyone else. It’s also rather disappointing that Dr. Spector would describe bevacizumab as a “nonspecific toxin” that interferes with blood vessel growth throughout the body. That’s not quite true. Bevacizumab does interfere with angiogenesis, but it tends to be much more specific for tumor blood vessels and other abnormal blood vessels than for normal blood vessels. Indeed, the most recent concepts regarding bevacizumab is that it actually “normalizes” tumor blood vessels, allowing chemotherapy to get to the tumor better. This observation explains a paradoxical observation that I made in the 1990s that radiation and antiangiogenic therapy with anti-VEGF antibody produced greater than additive anti-tumor effects.
Be that as it may, suppose I’ll be accused of “cherry picking” this example, but I don’t care that much. I choose it because I want to point out that the situation is not always as grim as Dr. Spector claims and that we have made enormous progress in treating some cancers over the last 40 years. I would also point out that the $105 billion we have spent on the war on cancer since 1970 breaks down to only around $2.6 billion a year. In recent years, the yearly budget for the National Cancer Institute has hovered in the $4.7 to $5.2 billion range. This may sound like a lot of money, but in reality it is only 0.1% of the $3.8 trillion federal budget this year. Let’s just put it this way, the entire yearly budget for the NCI would fund the war in Afghanistan for approximately 16 or 17 days. That is not a lot of money in the grand scheme of things; certainly it isn’t enough money to pay for a real “war” on cancer.
Where do we go from here?
Let me just begin my conclusion by conceding that Dr. Spector does make a few good points, albeit in insufficient detail in some cases. For example, his discussion of lead time bias is superficial and doesn’t even use the proper term for the concept, and he leaves out the concept of length bias in screening altogether. More irritating, though, is that there’s clearly a relentless effort on his part in both articles to paint as grim a view of the situation as before. Yes, Dr. Spector is correct that, if we could only decrease smoking to nothing we could decrease the death rate from lung cancer dramatically over the next 20-30 years. He’s also correct that decreasing use of HRT will help decrease the death rate from breast cancer, but in reality we’ve already done that. HRT use has declined precipitously after 2002; there isn’t much room to decrease it further. He also recommends vaccines to prevent HPV infection and thereby decrease the risk of cervical cancer, but that is really a pretty small contribution to the overall cancer death rate. Even so, let’s say we do all these things that Dr. Spector suggests because they are good things to do and will have an effect. Ignore for the moment how difficult it is to get people to stop smoking to prevent lung cancer, to stop drinking to prevent head and neck, esophageal, and liver cancers, and to lose weight to decrease the incidence of obesity-related cancers. Let’s say we can do it. What then? We will still have lots of people who will develop cancer and die from it if we don’t come up with better therapies. How do we do that? On that score, Dr. Spector is profoundly self-contradictory. I’ll show you what I mean.
At one point, Dr. Spector opines:
In my view the principal problem is that we just do not understand the causes of most cancers. We don’t even know if the problem is genetic or epigenetic or something totally unknown. In theory, problems 2 through 6 in table 6 are all correctable with political and scientific will and more knowledge. Even though we know cancer of the lung is caused by cigarette smoking, we do not know the mechanism, and (except for surgery) we do not know how to meaningfully intervene (see table 2). The pharmaceutical industry cannot make real progress until we understand the mechanisms and molecular causes of cancer so that industrial, academic, and governmental scientists have rational targets for intervention. We will make no progress if there are five hundred or more genetic abnormalities in a single cancer cell. Where would one begin?
And:
Moreover, with better mechanistic understanding of cancer, we could make truly “smart” drugs, as has been done in recent years for atherosclerosis (heart attacks), hypertension (strokes), gastrointestinal diseases (ulcers), and AIDS—with truly remarkable results. Let us hope cancer is next.
I can’t argue with that. Indeed, I’m very much about trying to understand the biology of cancer much better, so that we can design treatments that will be much more effective, “truly smart drugs,” as Dr. Spector puts it. That’s the direction, in fact, that I’m trying to move my research effort, with the help of a very good systems biologist. Here’s the problem. In his most recent article, Dr. Spector dismisses as useless and deadly the very techniques that we will require in order to generate the hypotheses that could lead to that better understanding of cancer upon which Dr. Spector’s proposed next generation of “smart drugs” would be based. Yet Dr. Spector poo-poos genome-wide association studies (GWAS), using them as an example of Deadly Hypothesis One. He lambastes the concept of personalized medicine and the genetic studies that will be necessary to achieve it as Deadly Hypothesis Six. He praises the methods used by the FDA, but, as explained by Mark Crislip, the rigor demanded by the FDA is the final step in a long process that begins with basic science, progresses through epidemiology and pilot studies, and then finally concludes with large, phase III randomized trials. Moreover, the final studies leading to FDA approval really are the “safest” part of the process, based on scads of data from all those earlier studies.
In other words, Dr. Spector, while claiming we need new and innovative hypotheses and ways to understand biology is in reality advocating very safe science indeed.
Dr. Spector makes a point that we need a better understanding of cancer, but he clearly has little idea about how to bring that about from a basic science perspective. So boxed into his clinical trial and pharma perspective is he that he castigates the very means of developing new hypotheses to test that we need. While Dr. Spector is correct that GWAS and gene expression profiling studies have been disappointing thus far, one needs to remember that the technology to do these studies has been in existence only a relatively few years and that these studies generate huge masses of data that strain even the best computers to analyze. In other words, we are early into these processes, but, as the cancer genome anatomy project demonstrates, we are learning.
That’s why I conclude that the worst aspect of Dr. Spector’s most recent article and his whole attitude in general is how he labels as Deadly Hypothesis One the concept that either “the investigator does not need a specific hypothesis and/or can use an inadequate method to test the hypothesis.” Yes, failing to use a specific hypothesis is not a good idea later in the process of drug development, but if we are to understand the biology of cancer sufficiently to design the next generation of “smart drugs,” we need new hypotheses, and these come from pilot studies. We sometimes call these studies “fishing expeditions” or, less pejoratively, “hypothesis generating experiments.” Come to think of it, one wonders whether Dr. Spector approved of the Human Genome Project. After all, there was no hypothesis to test there; scientists were simply trying sequence the entire human genome, and they learned a lot. More importantly, the results of the HGP generated many new hypotheses to test, many of which are being tested now, along with a revolution in genomic technology. The problem with Dr. Spector’s point of view is that he strikes me as seeing everything through the lens of pharma and the FDA. Scientific rigor is a good thing before approving drugs and treatments; in fact, it’s absolutely essential. However, the discovery process often begins by looking around and seeing if there’s anything interesting to see. Without that discovery process, all those rigorous trials meeting FDA requirements would never come to fruition because there would be no basic science pipeline to supply the translational research pipeline that eventually results in new drugs. Many–the vast majority, actually–of these hypotheses will never pan out, but they are just as essential to science-based medicine as the studies Dr. Spector loves.
Can we do better in the “war on cancer”? Of course we can–and must. However, it must be remembered that the “war on cancer,” like most wars, is not World War II. There is not, nor will there be, a total victory, an atomic bomb-like magic bullet that destroys all cancer. There will, as with most wars, be equivocal results, wins and losses, and pyrrhic victories. But we won’t get even that far if we embrace scientific nihilism over healthy skepticism.

   The Bait and Switch of Unscientific Medicine
Steven Novella 
Link
Savvy consumers are familiar with the classic scam of the “bait and switch” – in practice if not the term itself. My wife and I ran across it when we were shopping for our first car. We needed a bargain and so we were attracted to the ads that promised a new Colt for only $9,000 (that’s the bait). Of course when we got to the dealership they were all out of Colts with the configuration advertised, but they had plenty of others that had different options that cost several thousand dollars more (that’s the switch).
It’s a basic and very successful form of deception, and so even though there are laws against such practices it is impossible to eliminate in all its various and more subtle forms. It even permeates scientific, political, and other intellectual endeavors – anytime a more palatable idea or claim is put forward to represent the less acceptable truth.
Science, however, requires transparent honesty to function properly, and therefore scientific practitioners must vigilantly guard against the cognitive bait and switch. Generic intellectual virtues incorporate this vigilance – they include the need to unambiguously define terms, to make claims as specific and operational as possible, and the use of valid logic. Beware of any claims that subtly violate these rules because they are probably setting you up for a bait and switch.
The purveyors of unscientific medical claims have become as expert at this classic deception as the slickest used-car salesman – in fact they have left the hawkers of dubious transportation in the dust.
Practitioners of health fraud, unscientific and sectarian medical practices have learned how to apply the bait and switch at multiple levels – with respect to specific modalities and at higher categorical levels. First I will address some of the more common sectarian practices and show how they apply this deception, whether consciously or out of intellectual sloppiness.
Chiropractic
Chiropractic is perhaps the most common and egregious example of the bait and switch in medicine. The deception begins with the name itself – “chiropractic” fails the basic test of transparency because it is not unambiguously defined. There are in fact numerous professions doing very different things and employing mutually exclusive philosophies under the banner of “chiropractic.”
Therefore someone may go to see a chiropractor and think they will be seeing a medical professional who will treat their musculoskeletal symptoms, but in reality they will see the practitioner of a cult philosophy of energy healing. So-called “straight” chiropractors (who make up an estimated 30% of all chiropractors) still adhere to the original philosophy of chiropractic invented by “magnetic healer” D.D. Palmer, which is based upon the claim that an undetected life energy called “innate intelligence” flows through the spinal cord and nerves and is responsible for health. Such chiropractors will treat any disease or ailment with spinal manipulation.
Most other so called “mixer” chiropractors reject the notion of innate intelligence either partially or entirely, but still incorporate other pseudosciences into their practice. Chiropractors are the primary practitioners of homeopathy, applied kinesiology, and iridology in the US. The bait – claims that chiropractors are medical practitioners with expertise in the musculoskeletal system. The switch – practitioners of discredited pseudosciences that have nothing to do with the musculoskeletal system.
A more subtle form of the bait and switch among chiropractors is the treatment of musculoskeletal symptoms with standard physical therapy or sports medicine practices under the name of chiropractic manipulation. Ironically, the more honest and scientific practitioners among chiropractors are most likely to commit this subtle deception. The problem comes not from the treatment itself but the claim that such treatments are “chiropractic.”
Using techniques like massage, range of motion exercises, strength-building exercises, and mobilization of joints are all legitimate science-based techniques used by physical therapists and physicians with specialties in physiatry, orthopedics, and sports medicine. Some chiropractors also use similar techniques -and with good results. But by doing so and calling it “chiropractic” it legitimizes the pseudoscientific practices that are very common within the profession – like treating non-existent “subluxations” in order to free up the flow of innate intelligence.
Acupuncture
Many studies of acupuncture employ this latter form of bait and switch – studying something that is not really acupuncture but calling it acupuncture and thereby legitimizing pure pseudoscience. Acupuncture refers to a Traditional Chinese Medicine (TCM) pre-scientific philosophy of health and illness that is based (very similarly to straight chiropractic) on the flow of life energy called “chi”. Fine needles are inserted into special locations on the body called meridians for the purpose of improving, freeing, and balancing the flow chi.
No study has demonstrated that chi exists, or that acupuncture of the meridians has any specific effect. A look at the entirety of the acupuncture literature, in fact, clearly demonstrates that it does not matter where the needles are placed (or how deeply, or if they are manipulated in ways that are designed to manipulate chi). This means that any effects of sticking acupuncture needles into a patient are non-specific – they are not related to the flow of chi. There may be some small non-specific physiological effects – such as counter-irritation reducing pain or inhibiting nausea – but even these claims remain elusive and controversial.
But some studies of “acupuncture” show a larger and more reliable effect, usually on pain. What these studies have in common is that they employ “electrical acupuncture” (for example). Electrical acupuncture, however, is not acupuncture – it’s transdermal electrical stimulation, which is a scientific practice that has proven efficacy in the treatment of pain. Giving transdermal electrical stimulation through acupuncture needles, calling that “electrical acupuncture” and then using positive results to conclude that acupuncture works – is an elaborate bait and switch.
Homeopathy and Herbal Medicine
Homeopathy is another dubious practice based upon pre-scientific notions long discredited. A truly homeopathic remedy is diluted to the point that no active ingredients remain – so a homeopathic remedy is pure solvent, either water or alcohol, sometimes placed on a sugar pill (making a homeopathic pill a literal placebo). But much of the public believes that “homeopathic” simply means “natural” or “herbal”. Few realize that it means that the so-called medicine contains nothing.
But some homeopathic products are not even truly homeopathic – meaning they are not based upon homeopathic principles but rather contain measurable amounts of real drugs. This is now not uncommon for over-the-counters lotions and cold remedies. For example, this pimple cream contains many active ingredients at a “1x” dilution – that’s only 1:10, meaning there is still actual molecules of drugs in the cream. That’s not homeopathic.
A similar and more subtle deception is used in the marketing of herbs. They are sold and regulated as “natural supplements” when in fact they are drugs. Herbs contain chemicals that have pharmacological activity and they are usually taken in doses that are insignificant in terms of their nutritional content, but are significant in terms of their pharmacological activity. The bait is that they are nutritional supplements, the switch is that they are unpurified drugs.
CAM – the Biggest Bait and Switch
All of these modalities fall under an artificial category – so called “complementary and alternative” medicine (CAM) or “integrative” medicine that was created as the ultimate bait and switch.
The deception is largely two-fold. The first is to include modalities within this false category that are legitimate, like nutrition, exercise, physical therapy, relaxation, etc., – and then claim that these legitimize the entire category of CAM, even the far-out stuff like homeopathy. This is just a higher-order version of including physical therapy modalities within the umbrella of chiropractic, for example.
Essentially, at every level terms are used that are vague and poorly defined, and in practice are designed to be all-inclusive so that the purely pseudoscientific can ride on the coattails of the barely plausible or even the legitimate.
This is the opposite of what happens within scientific disciplines – where terminology evolves over time to become more discriminating, unambiguous, and specific. To take an example from my specialty, multiple sclerosis (MS) has been divided into sub-diagnoses: relapsing remitting and chronic progressive. Divisions multiplied and now there are seven subtypes – for example, primary progressive, and progressive remitting.
This increased “splitting” as it is called is not just a pedantic dedication to precision, but a recognition that such divisions may reflect differences in underlying mechanism of the disease. Also, different subtypes of MS respond differently to different treatments. Some drugs work in relapsing remitting but not in primary progressive. Part of the motivation for finer categories is to more accurately predict which MS patients are likely to respond to which treatments.
Contrast this to the maximally broad and imprecise but common marketing slogan “chiropractic works.”
The second kind of deception created by the category of CAM is in the language used itself – “complementary” and “integrative.” What, exactly, are CAM modalities integrating? On close examination it is quite clear – the movement is an effort to mix unscientific, disproved, and dubious modalities into scientific medicine. The bait is that CAM offers legitimate alternatives, the switch is that it primarily promotes treatments that don’t work or are at best untested and highly implausible.
If there were truth in marketing then we would have the Office of Implausible Medicine, the Journal of Bad Medical Science, the Center for Rejected Therapies, and the Institute of Dubious Medical Claims – all under the umbrella of unscientific medicine. It used to be called, even more simply, “health fraud.”
Conclusion
The public would be best served if they were given a reasonable expectation of honesty and clarity in the health marketplace. Instead, increasingly they are being given the old bait and switch, and increasingly the governments and institutions of medicine that should know better are instruments of this deception.
Science and intellectual honesty require that terms mean something specific and definable. Unscientific health modalities have become very savvy at using terminology to confuse rather than inform, to deceive rather than to illuminate.
It is the simplest form of the bait and switch – just call a product or service something that it isn’t.

   Pragmatic Studies – More Bait and Switch
Steven Novella 
Link
The course of research into so-called alternative medicine (CAM) over the last 20 years has largely followed the same pattern. There was little research into many of the popular CAM modalities, but proponents supported them anyway. We don’t need science, they argued, because we have anecdotes, history, and intuition.
When media attention, which drove public attention, was increasingly paid to CAM then serious scientific research increased. A specific manifestation of this was the National Center for Complementary and Alternative Medicine (NCCAM). CAM proponents then argued that their modalities were legitimate because they were being studied (as if that’s enough). Just you wait until all the positive evidence comes rolling in showing how right we were all along.
But then the evidence started coming in negative. A review of the research funded by NCCAM, for example, found that 10 years and 2.5 billion dollars of research had found no proof for any CAM modality. They must be doing something wrong, Senator Harkin (the NCCAM’s major backer) complained. They engaged in a bit of the kettle defense – they argue that the evidence is positive (by cherry picking, usually preliminary evidence), but when it is pointed out to them that evidence is actually negative they argue that the studies were not done fairly. But then when they are allowed to have studies done their way, but still well-controlled, and they are still negative, they argue that “Western science cannot test my CAM modalities.”
But at the same time, they cannot shake the need for scientific evidence to continue to push their modalities into the mainstream. The “your science can’t test my woo” defense only goes so far. So they have put themselves into a pickle – they demanded funding for CAM research, but now have to deal with the fact that the research is largely negative. CAM proponents are mostly not interested in finding what really works, and abandoning what does not work (I have an open challenge to anyone who can point to a CAM modality that was largely abandoned, rejected, or condemned by CAM proponents because of evidence of lack of efficacy). They are interested in using science to support and promote what they already believe works (a cardinal feature of pseudoscience).
And so they have entered the next phase of CAM research – the post-RCT (randomized controlled trial) phase. They have discovered the “pragmatic” study. You have to give it to them for their cleverness. A pragmatic study is meant to be a comparison of treatment options in real-world conditions. It studies treatments that already have proven efficacy from RCTs to see how they work, and how they compare, when applied in the less controlled environment of real-world practice. Pragmatic studies are useful for addressing the weaknesses of RCTs, mainly their somewhat contrived conditions (having strict inclusion criteria, for example).
But pragmatic studies are not efficacy trials themselves. They cannot be used to determine if a treatment works, because they do not control for variables and they are not blinded, so they are susceptible to placebo and non-specific effects. It is an abuse of the pragmatic study design to test a treatment that is not proven or to make efficacy claims based upon them. That has not stopped CAM proponents from doing just that.
In the news recently is just the latest example. Let me tell you the results of the study before I tell you what the study is of – just look at the data (I replaced the name of the treatment in the tables with just the word “treatment”):
Link to graphic
This is what you need to know about the trial. There were two groups, treatment and control. Subjects were those with frequent doctor visits for symptoms that have not been diagnosed. While they were randomized, they were completely unblinded – everyone involved with the subjects knew who was getting treated and who wasn’t. At 26 weeks the control group was crossed over to receive treatment – so the difference up to 26 weeks is most important.
On two of the measures, quality of life and general practitioner consultation rates, there was no difference. In the other two measures there was a slight improvement in the treatment group, barely statistically significant. If this were an efficacy trial, this data would be unconvincing. What can fairly be concluded from this trial is that the treatment has no to minimal effect, and the tiny and inconsistent effect seen cannot be separated from placebo or non-specific effects. Further, it shows how anemic even placebo effects are for this treatment in this patient population. As a pragmatic trial, it’s essentially negative. As an efficacy trial, it’s worthless.
The study author’s, however, concluded that their treatment was effective and recommended it be incorporated into general practice.
The treatment in question is five-element acupuncture – acupuncture designed to balance the five elements of fire, water, metal, earth, and wood. This is the equivalent of balancing the four humors – in other words, it is pre-scientific superstitious nonsense. The study authors are all proponents. What is most amazing is how they got their paper, complete with unsupported conclusions, past peer-review.
The study authors are using the “part of this nutritious breakfast” con. Even as a child I understood that the sugary cereal, or whatever the commercial was trying to sell to me, was only “part of this nutritious breakfast” because the rest of the breakfast was nutritious all by itself. A doughnut would be “part of this nutritious breakfast.” So the authors of this study include five-point acupuncture with an hour of kind attention from a practitioner along with lifestyle advice and encouragement. They admit that they cannot separate out the variables here. Then they conclude that acupuncture is “part of this healthy regimen.” There is every reason to believe that it is an irrelevant part – as irrelevant as the doughnut is to the nutrition of a complete breakfast.
Conclusion
The pragmatic study bait and switch is here – it is now a firm part of the CAM strategy for promoting implausible therapies that don’t work. Editors and peer-reviewers need to be more aware of this scam so they don’t fall for it and inadvertently promote it, as they did in this case. Further, the results of this trial are not impressive even at face value – it’s basically negative, so it’s a double swindle.
All of this has not stopped the headlines from declaring that “acupuncture works” – which appears to be the only goal of such research.

   Deadly Indeed
Mark Crislip
Link
There are sources of information I inclined to accept with minimal questioning. I do not have time to examine everything in excruciating detail, and like most people, use intellectual short cuts to get through the day. If it comes from Clinical Infectious Diseases or the NEJM, I am inclined to accept the conclusions without a great deal of analysis, especially for non-infectious disease articles. Infectious disease publications I have to read more closely; its part of passing as an expert.
Outside of medicine, I am predisposed to accepting at face value many of the articles in Skeptic and Skeptical Inquirer. They are trusted sources. Some topics, like haunted house or Big Foot investigations, I barely skim. After all these years, I doubt there will be any new insights into the subject. Other topics, depending on my interest, I may read more carefully.
I often read longer articles many times. First a quick skim to see if it offers anything of interest. If it does, then I may read it carefully.
This months Skeptical Inquirer had an article called Seven Deadly Medical Hypotheses by Reynold Spector. Just seeing the title and knowing the magazine, I was primed to accept the content at face value. I enjoy a well reasoned, thoughtful rant. I relish a clever diatribe, even if I do not agree with the topic. So I gave it a quick skim. I was discomfited. My first gut check was ick. But I was uncertain why. So I read it slowly and carefully, and still ick. But why?There is a degree of self absorption in being a blogger. I can write about what I want any way I want (I remain amazed at how much I can get away with). The process of writing about a topic helps me clarify in my own mind issues with articles.
The author of 7, as I shall refer to the article, has over 200 published articles, is a former executive vice president in charge of drug development at Merck and oversaw the development of 15 drugs and vaccines. I am nobody from nowhere who just takes care of infected patients for a living. He wins the argument from authority; I am the E. coli evaluating the human. Oh well, this is more an exercise for me to enlighten myself; you are the innocent bystander.
Overall the tone of 7 ? It reminded me of the Health Ranger at NaturalNews.com. Really. Lots of dramatic statements, no qualifiers, no buts, no subtlety, no nuance. To me, what marks good medical writing is an understanding that there is far more grey than black and white and that generally people are doing the best they can within numerous limitations One of the many characteristics of the Health Ranger is hyperbole without nuance. The Health Ranger has a belief system and sees the medical industrial complex through that lens; information is used to support a predetermined conclusion. Health Ranger is a bombastic style that is both self assured and self referential.
Let us see what 7 has to say. It begins
A chronic scandal plagues the medical and nutritional literature: much of what is published is erroneous, pseudoscientific, or worse.
I’ll grant the first. I am an Ioannidis convert. The second seems hyperbole and exaggeration. Pseudoscientific? Like homeopathy, psi and astrology? Sorry The author is 17 words in and he has lost me. I already question his veracity and judgement. I read the literature. Hundreds of papers a month. I know the literature, and Sir, it is not pseudoscientific. Suboptimal, often, but not pseudoscientific. The third? What could be worse than pseudoscientific? Oh yeah. Wakefields Lancet article. But fraud is a very rare exception in the over 20 million references on Pubmed. The author’s opening salvo strikes me as someone more interested in polemic than truth. If done with verve and panache, and above all wit, I like a good polemic. Pomposity with hyperbole, not so much, and calling the medical literature erroneous, pseudoscientific, or worse leans towards the latter.
Two major factors account for a large proportion of this problem. First, many medical and nutritional hypothesis are ill-conceived.
Are they? Over 20 million references in Pubmed. A few, perhaps, were ill conceived before they are tested. Say, measles vaccine induced gastroenteritis causing autism? Not even that. If approached honestly and competently, it would be a long shot, but you never knows unless you look. That is what a great deal of medical research is about: looking around to see if an etiology or intervention or medication will be effective. Most ideas, I would guess, go nowhere.
Second, the methods used are often epistemologically unsound.
Got me there. What is epistemologically unsound? Even after looking epistemologically up on the interwebs, I am uncertain what it means. I expect the comments will school me on the meaning of epistemologically unsound. I guess that is why I am a lowly clinician.
Moreover, the same unsound methods are often repeated multiple times on the same tired hypotheses with the same incorrect results.
Isn’t that three major factors? Or is that the unsound epistemological I cannot understand? I shouldn’t quibble about counting, but I feel a rising tide of ridicule and scorn, and I am not one to hold it back.
I am not even done with the first paragraph, and the author has epistemologically lost me. Maybe there is good reason to be unsettled with the article. And in the first five sentences, there are four references, all to works by the author, to justify the position. I tend to prefer external references in my literature; the hyperbolic self-validation is what I expect from the Health Ranger and his ilk. But again, who am I to question (1)?
… there is an epidemic of published studies that do not follow the principles of sound medical science- the principles demanded by the US Food and Drug Administration for the licensure and sale of medications.
Well, most studies are preliminary and exploratory. The rigor demanded by the FDA is the final step in a long process starting with basic principles and, perhaps, epidemiology. I can’t imagine we should jump to huge randomized, placebo controlled trials for every therapy and to answer every question. Seems a wee bit excessive to me. Start small and build. The downside is that there will be dead ends and false conclusions. The upside is that in the end, a close approximation of Truth will be determined.
The resulting “findings” of such misleading or erroneous studies are often hyped by the news media on the day they are reported or published without any additional, careful analysis.
Hyped “findings”? Nothing like that in the first two paragraphs of the this essay. Nope. Nothing to see there but a well reasoned, careful, nuanced prologue for the body of the essay. “I” am always “mistrustful” of people who use “quotes” as a form of “sarcasm” when sarcasm is not used for good “effect” like “humor” because it otherwise comes across as “supercilious”.
Now I am starting to understand my discomfiture. Still, that’s just the first two paragraphs. The body will better, right?
The author then proceeds to the background of how to do a good study: generate a plausible, testable hypothesis and test it. He uses the Scandinavian Simvastatin Survival Study as an example of medicine done right (a Merck product if you care) and bemoans that not every study meets this high standard.
Too many published studies fail to adhere to these high scientific standards and lead to faulty, and even dangerous, conclusions.
Which is true and to my mind understandable, since there are not the resources to do perfect studies of every hypothesis. Not every car is a Lexus, not every restaurant has a Michelin three star rating. You can’t always get what you want (2). The issue to my mind is not that there are suboptimal studies; they are often used to find search for hypothesis that can be tested in better trials. A large part of research is flailing about looking for something interesting to investigate in further detail. Not everyone has the resources to test everything using the ‘hypothetical/deductive method” to answer all out questions, like the FDA demands. Although this is not always the preferred method of generating ideas to test. I don’t need quotes to cast aspersions on the validity of information or generate guilt by association. I have learned a thing or two from reading the Health Ranger.
I wonder how many suboptimal studies it required to get to the point of the Scandinavian Simvastatin Survival Study? The concepts to be tested did not appear from the void, fully formed. The author does not, as will be seen, pay attention to the history and context of the evolution of medical ideas.
The author then proceeds to his 7 deadly hypotheses. Well, one deadly, 6 not so much. But guilt by association is a game played by the author of 7 as well.
1) the investigator does not need a specific hypothesis and/or can use an inadequate method to test hypothesis.
He uses the example of epidemiology generated by case-control and cohort studies (the kind of studies that lead to the simvastatin study) and the effects of hormone replacement therapy. He points out that these epidemiologic studies, for a variety of reasons, can lead to erroneous conclusions. Fine. The other option? With no preliminary studies jump straight to a huge trial? And sometimes epidemiology can lead to important results: that a certain water pump is the epicenter of cholera or that chimney sweeps have more testicular cancer. Or that lowering cholesterol is associated with a decrease in vascular deaths.
Epidemiology is part of a continuum of understanding and evolution of medical knowledge. But strawmen are easier to burn than recognizing the stuttering, somewhat chaotic progress of medical knowledge. If proving a point is more important than understanding complexity, this is how you argue.
He then proceeds to genome-wide association studies (GWAS) that have been a disappointment for elucidating genetic causes of heart disease and Alzheimer’s. The author considers GWAS a failure. I suppose if you have a narrow perspective, yes it has been a failure. So far. Huge amounts of information about the genome have been generated, and I am always a fan of knowledge for knowledge sake. In the world of infectious diseases, there are single gene polymorphisms in the immune system that can increase or decrease a patients risk for a variety of infections. Is it of clinical relevance yet? No. Is it interesting? Oh, yes. Will it lead to a new treatments and diagnostic interventions in the future? Who knows. But trying new ideas may fail but still lead to insights that may lead to better interventions. I would wonder what secondary advances in technology and understanding were accomplished as a results of the GWAS studies.
It is like complaining that the Apollo program only put 12 people on the moon so the program is a bust since we are not going to the moon for vacation. Here is a dirty little secret from a mere clinician. I learn far more from failure than I ever have from success. “The most exciting phrase to hear in science, the one that heralds new discoveries, is not ‘Eureka!’ (I’ve found it!), but ‘That’s funny…’ -Isaac Asimov.” If you are a clinician, it is not ‘That’s funny,’ but ‘Oh shit’ that really drives change and knowledge.
2) If women replace these missing hormones post menopausally with HRT, they will remain “youthful” and not suffer from heart disease, dementia, vaginal dryness, hot flashes, and fractured bones.
I remember the late ’80′s, a time that was the heyday of HRT, when I was in my internal residency training and discussing the issues at length not only in clinic, but with my mother. I remember discussing the epidemiologic data and the worries of cancer. The author states that
…based on these (biased) studies, false claims were made the HRT protected against cardiovascular disease and dementia.
As if we knew it was false at the time. It was the best guess based on the data, and epidemiology can give insights that can be later confirmed by better studies. He also says
“the proponents…ignored the well-documented fact that estrogen is a carcinogen that causes breast cancer that can kill women” and that “HRT caused a 25% increase in breast cancer.”
I do not know where the author was practicing, but I remember talking with patients (I know, flawed memory) and my mother about the relative risks of cancer and fracture from HRT. And 25%. Increase. That’s bad.
What was the study? On “16 608, patients, there were more invasive breast cancers compared with placebo (385 cases [0.42% per year] vs 293 cases [0.34% per year]…and the estrogen group had higher mortality (25 deaths [0.03% per year] vs 12 deaths [0.01% per year].”
That is bad. Equally bad was the way the author presented the data, the same author who complains in the opening paragraphs about complex data being presented as looking “superficially adequate to the unsophisticated reader,” but I know when someone is presenting information in an manipulating manner designed to blow smoke out a usually inaccessible area.
In a section worthy of the Vaccine Council or Dr. Mercola, it sounds like people deliberately ignored cancer risk to push estrogen to kill women. Someone mention hype? I know it is important to make a point, but those who were investigating HRT and prescribing it, as I did once upon a time, were doing it carefully and with knowledge that there could be risks.
Information does not exist in a vacuum. When talking with my patients and Mom in the late 1980′s, I basically said, based on the odds, how do you want to live your life?
Lifetime risk is a useful way to estimate and compare the risk of various conditions. Hip fractures, Colles’ fractures, and coronary heart disease, and breast and endometrial cancers are important conditions in postmenopausal women that might be influenced by the use of hormone replacement therapy. We used population-based data to estimate a woman’s lifetime risk of suffering a hip, Colles’, or vertebral fracture and her risk of dying of coronary heart disease. A 50-year-old white woman has a 16% risk of suffering a hip fracture, a 15% risk of suffering a Colles’ fracture, and a 32% risk of suffering a vertebral fracture during her remaining lifetime. These risks exceed her risk of developing breast or endometrial cancer. She has a 31% risk of dying of coronary heart disease, which is about 10 times greater than her risk of dying of hip fractures or breast cancer. These lifetime risks provide a useful description of the comparative risks of conditions that might be influenced by postmenopausal hormone therapy.
That was the kind of information and conversations about HRT I was having with patients in my clinic as I completed my residency, the years the author was at Merck developing drugs. Many patients were far more worried about the disability and pain of fractures than they were of breast cancer.
In continued hyperbole that is totally disconnected from what I remember, he calls HRT a “flagrant example of the harm done by straying from the principles of hypothetical/deductive approach and sound clinical science.”
Really? Did this guy ever take care of patients? Has he ever had to make decisions based on incomplete information? We are only into number two of seven and he last lost me with the hysteria. I wonder how he would suggest exploring the effects of waning estrogen on the health of women? Jump straight to a large trial? Do no preliminary work? Ignore any potential leads? What is the alternative to the incremental, and sometimes erroneous, results of medical understanding? How about fluoride and tooth decay? So many insights start with a guess and a little epidemiology. Sometimes it pans out, sometimes it doesn’t. But you do not know unless you try.
3) if small dosages of vitamins are good for humans, very large doses would be better for everyone.
He then notes the studies that show the hypothesis was wrong. But this was only known after the fact, after the studies, and perhaps using vitamins like drugs would have beneficial effects.
Then the odd summary: ” megavitamin therapy tested in properly controlled trials either does nothing or is harmful (except in a few well defined exceptions).”
So it does nothing except when it does. And how would we know the well defined exceptions unless we did the trials?
He goes from complaining about the science to complaining about the regulatory and commercial issues of megavitamins, changing arguments in midstream. Is it the science or how the science is used? Two different issues.
This is getting tedious, even for me. I will soldier on, although the re re re reading 7 is increasingly painful. The closer I read it, the greater the errors and manipulations; a Mandelbrot set of manipulative medical writing. Soon I will find the indefinite articles and pronouns suspect. I try to skim the Health Ranger for a few chuckles; that is not why I read SI. And when is their swimsuit issue? Oh. Wrong SI.
4) Screening tests beyond the standard medical examination are necessary for identifying disease and the risk of disease in apparently healthy, asymptomatic adults.
I will leave this issue to the more knowledgeable hands of Dr. Gorski. His argument seems to be based on the 20:20 vision of hindsight, which is apparently the primary argument in all seven cases. We thought screening would be effective, studies showed it wasn’t, so the hypothesis was flawed and we should not have suggested screening or done the studies.
The author does not show in this, or other examples, why the ideas were wrong in the context of time the ideas were first offered. It is only viewed through the all powerful retrospectoscope that the author finds his deadly hypothesis. It is ever so easy to predict the past.
He also seems to argue that since our understanding of the ramifications of screening are not perfect, they are suspect, referencing himself for issues with PSA and mammograms (1). The author argues in part that since our understanding is imperfect, it is a deadly hypothesis. I have always been comfortable with making decisions based on incomplete information, as that is the only kind of clinical information we ever have, save for the results of the occasional autopsy. The perfect always being the enemy of the good.
He also complains about genetic screening. He notes that few people with high risk genes will develop disease and they can’t do anything about it, so why bother? I wonder if the author has had much direct patient care. What most patients dislike is uncertainty about the why of their disease and most prefer as much understanding and certainty about their health as they can gather. That is why they bother. And todays why bother may be tomorrows critical insight. I have discussed how the show Connections made an impact on my view of the serendipity underlying advances. It may not be cost effective or useful currently, the author does note that for some patients (breast cancer) it may have utility. Again, it is a deadly hypothesis except when it isn’t. So much sound and fury.
But how do you know until after you have done all the studies and see what works and what doesn’t? His argument still seems to be since in some patients genetic testing has been shown to be of no utility, in the past they should not have done the work to show it is not useful. Except where it is. Sort of like going back in time to kill Hitler as a child because he was found to be evil in the future, even though you could not tell that the babe in the crib was going to be the source of Goodwin’s law. And far worse.
Circular argument much?
I do not get the impression the author is one for thinking outside the box. Usually new ideas lead nowhere, but again, you never know unless you try. Nothing ventured, nothing gained vrs nothing ventured, nothing lost. It is often not the results of studies that are the issue, but how they are portrayed in the media, as noted by the author, and, probably not intentionally, his entire article is a superb example of just that concept. Maybe 7 is really meta.
5) Manipulating one’s nutrition can prevent cancer.
As he says, “In retrospect, this hypothesis does not seem plausible.”
The whole crux of almost every one of his arguments. Repeat after me. In retrospect. In retrospect. In retrospect. In retrospect everything is clear. I have had MD after my name for 27 years, and I remember the uncertainty and interest in all his 7 mostly not so deadly hypotheses. In the beginning, it was not so clear as he makes it out to be. The past is easy to predict.
6) Personalized medicine will greatly advance medical care.
His argument is the same: it hasn’t worked except where it has.
“Personalized medicine has only been shown to be cost effective in a few well defined situations.”
How did we find these well defined situations? Doing a ton of studies that show benefit in some cases and none in others.
I think the solution to this problem is being able to see the future and know in advance which research ideas will bear fruit and which will be a bust. Precognition is apparently the only solution. Miss Cleo may be available to help review research proposals, I understand that her readin’ is free.
7) cancer chemotherapy has been a major medical advance.
Of course, in some cases it has been extremely effective, but the war on cancer has not been what it was promised. Again it seems his argument is the same hindsight argument: when cancer therapy has been effective, it is great, and when it is not so good, we should not have done the work to show that wasn’t effective. Again, I leave the details to Dr. Gorski should he choose to cover the topic.
And of course the author doesn’t have a dog in the fight (and there are those quotes, so commonly used by the dispassionate):
“When one dispassionately weighs the minimal prolongation of ‘good’ life in patients with metastatic cancer versus the very distressing side effects of chemotherapy with ‘targeted’ drugs, the case is close.”
I’m convinced, He is dispassionate. And Jenny isn’t anti-vax, just pro-safe vaccine. Here is my hypothesis to be tested. Anyone who argues they are dispassionate isn’t. They are fooling themselves and trying to fool others with their alleged practice of arei’mnu. Me? I am never dispassionate; although sometimes I do not care, but there is a difference.
Some of his conclusions are reasonable: we need to do our science as best as we can.
The author argues that all these errors and expenditures of his 7 mostly no so deadly hypotheses could have “been avoided if the hypothetical/deductive method had been applied rigorously.” I am not convinced, since most of his arguments are based after the fact. I would be far more impressed if, by using only the hypothetical/deductive approach (no epidemiology, no early studies, no preliminary clinical data, no basic science) if he would predict 7 hypotheses that warrant jumping straight to large, randomized, placebo controlled clinical trials so beloved by the FDA. The Randi prize awaits.
We all need that god like perfection and prescience, unlike those
“guilty of perpetuating worthless practices include “scientists” who repeatedly employ flawed methods and then publish them, government agencies who fund such practices, editors of journals that publish pseudoscience, the USDA and NCI bodies that perpetuate unscientific regimens…”
My. God. The Health Ranger was right. The conspiracy has incorporated itself into every aspect of the Medical-Industrial complex. A different conspiracy than the one we get from the woo world, but everyone is involved.
Putting scientists in quotes. A very Health Ranger thing to do. I don’t suppose he is referring to the “scientists” at Merck who repeatedly employed flawed methods and then published them.
“Approximately 250 documents were relevant to our review. For the publication of clinical trials, documents were found describing Merck employees working either independently or in collaboration with medical publishing companies to prepare manuscripts and subsequently recruiting external, academically affiliated investigators to be authors. Recruited authors were frequently placed in the first and second positions of the authorship list. For the publication of scientific review papers, documents were found describing Merck marketing employees developing plans for manuscripts, contracting with medical publishing companies to ghostwrite manuscripts, and recruiting external, academically affiliated investigators to be authors. Recruited authors were commonly the sole author on the manuscript and offered honoraria for their participation…
This case-study review of industry documents demonstrates that clinical trial manuscripts related to rofecoxib were authored by sponsor employees but often attributed first authorship to academically affiliated investigators who did not always disclose industry financial support. Review manuscripts were often prepared by unacknowledged authors and subsequently attributed authorship to academically affiliated investigators who often did not disclose industry financial support.”
I see people doing the best they can with the tools at hand. Mostly honest people (I say mostly not knowing what their IRS forms show), working within many limitations, to advance medical understanding. They do not deserve quotes applied to their work or the title of pseudoscience. Not everyone is able to achieve the peerless, perfect knowledge bestowed on a Professor of Medicine and Merck Vice President.
We need “honest” corporations. Ironic from a former Merck executive; casting the first stone and all that. I do not need quotes to show my snotty superiority. We need better regulation of “unsafe and unproven products.” Like Merck’s Vioxx?. Ohhh, snap. The Merck shots are cheap shots, I know. But they made me laugh, and above all I like to make me laugh. It is all about me.
Like the Health Ranger, I see someone with a bee in their bonnet, selectively and histrionically arguing in circles, hoping that if the same cognitive errors and circular reasoning are repeated they will be believed as fact. I am not enthusiastic about the conclusions and arguments used, being significantly more flawed than the research he rails against. It is not far in style and content from being in the Natural News. Science, at least, is ultimately self correcting. This article, probably not so much.
Of course, I am nobody from nowhere. Not a professor or scientist or a vice president. I am a clinician and citizen who has to trust his sources of information. I was raised to judge a man by the company he keeps. When the NEJM published garbage on acupuncture, my trust in the Journal fell a notch. The Lancet has always had a reputation of being flaky, it is part of the British charm and I have never held it against them; I just factor it in when reading a paper. The Annals of Internal Medicine has been untrustworthy for years. Clinical Infectious Diseases remains unsullied. Now the Skeptical Enquirer (sic) has slipped a bit as well. 7 was primarily deadly for my confidence in its editors. Oh well, at least I can still trust the material published by DC.
Rationalizations
(1) Crislip et. al. I said it here before, so it must be right.
(2) And if you try sometime you find/You get what you need.

   Seven Deadly Medical Hypotheses revisited
David Gorski 
Link
Back in February, Mark Crislip and I both deconstructed an article written by Dr. Reynold Spector that appeared in the March/April issue of Skeptical Inquirer (SI), the flagship publication for the Committee for Skeptical Inquiry (CSI). The article was entitled Seven Deadly Medical Hypotheses, and, contrary to the usual standard of articles published in SI, it used a panoply of spin, bad arguments, and, yes, misinformation to paint a picture of seven horrifically deadly “medical hypotheses,” most of which, even if the reader accepted Dr. Spector’s arguments at face value in a worst case scenario, weren’t actually all that deadly at all, with the alleged deadliness of the others being in dispute. In addition, Dr. Spector painted a picture of medical science that is not nearly rigorous enough. While we at SBM would probably agree that much of medical science is insufficiently rigorous, given how so-called “complementary and alternative medicine” (CAM) or “integrative medicine” (IM) has found a prominent place in medical practice in all too many academic and private medical centers, Dr. Spector got it so wrong that he wasn’t even wrong when he conflated preliminary, hypothesis-generating studies with the big, randomized, phase III clinical trials necessary to achieve FDA approval for a new drug or device. This latest article by Dr. Spector seemed to be of a piece with his previous article in the January/February 2010 issue of SI entitled The War on Cancer A Progress Report for Skeptics, which was so negative in its assessment of scientific progress against cancer that for a moment I was wondering if I were reading NaturalNews.com or Mercola.com.
Unfortunately, Seven Deadly Medical Hypotheses itself is not yet online on the CSI website; so readers without a subscription to SI cannot at the present time judge for themselves whether Mark and I were too harsh on Dr. Spector, but our criticisms, along with that of SBM partner-in-crime Harriet Hall, did have an impact. Seemingly genuinely stunned at the level of criticism leveled at an article published in SI, SI’s editor Kendrick Frazier, to his credit, invited several responses to Dr. Spector’s article, which Harriet Hall, Mark Crislip, Carol Tavris, Avrum Bluming, and I eagerly provided. These letters were originally scheduled to be published a couple of issues ago, along with Dr. Spector’s response. Unfortunately, publishing in dead tree media being what it is, Harriet Hall and I were disappointed to find that the latest issue of SI still didn’t contain our rebuttals. Fortunately, Mr. Frazier has posted this material online for your edification, although, again, I wish he had also published the original article as well.
What other skeptics said about Seven Deadly Medical Hypotheses
Before I deconstruct Dr. Spector’s response to criticism, I think it’s worth briefly commenting on some of the other criticism that Dr. Spector garnered, the first of which appeared in the letters section of the July/August 2011 issue of SI in the form of two letters, one by Dr. James Kolter and one by a professor of social science named Robert J. Miller. Unfortunately, both appear to be primarily single issue critiques, rather than taking on the entire wrong gestalt of Dr. Spector’s article. For example, Dr. Kolter, as both Dr. Crislip and I did in our SBM blog posts, pointed out that the evidence indicting hormone replacement therapy (HRT) after menopause as a cause of breast cancer is not as definitive and clear-cut as Dr. Spector represented it, while Dr. Miller took Dr. Spector to task for a “seeming too-easy dismissal of a positive role of fresh fruit and vegetables in disease prevention.” Both of these are, of course, valid criticisms, but the miss the meta-issues underlying Dr. Spector’s medical pseudoskepticism, one of which Mark Crislip, in his usually insightful manner, picks up on right away:
The article “Seven Deadly Medical Hypotheses” has fundamental flaws. The argument, as I understand it, is that medical researchers fail to use the hypothetical/deductive approach to decide upon research agendas and therefore waste a “vast quantity of resource to disprove them.”
The argument as presented is circular. The author uses information from completed studies to prove the studies should not have been done in the first place. He fails to consider the state of knowledge at the time the studies were done and the cumulative nature of medical knowledge.
Meanwhile, Carol Tavris and Avrum Bluming pointed out a more irritating aspect of Dr. Spector’s entire attitude:
Of course, scientists and other skeptics can and do disagree with one another when it comes to interpreting data. Our primary objection to Spector’s discussion of HRT is not his conclusion but his tone of certainty, his oversimplification of an enormous body of research, and his unwillingness to question received wisdom from the WHI. Even some investigators who share his belief that the relative risks of HRT warrant concern acknowledge that the absolute risks from this treatment are small. In one worst-case analysis, researchers calculated that a fifty-year-old woman taking estrogen and progestin for ten years has a 96 percent chance of remaining free of breast cancer versus a 98 percent risk if she does not take HRT.
I will note that, as a breast cancer surgeon, I do not quite agree with Tavris and Bluming with their apparent glee in trashing the Women’s Health Initiative study that led to the publication of the famous paper in the Journal of the American Medical Association back in 2002 concluding that estrogen/progestin combined HRT increased the risk of breast cancer, a finding that Tavris and Blum apparently do not accept. Of course, this is an example of how scientists and other skeptics can disagree when it comes to interpreting data.
Finally, Harriet Hall, a.k.a. The Skepdoc, weighs in with a devastating characterization of Seven Deadly Medical Hypotheses:
Spector’s article was doubly disappointing because he had so many excellent points that were sabotaged by the way he presented them. Medical science is far from perfect, but it is a cooperative, self-correcting endeavor that constantly criticizes itself and is constantly improving. We need continued rational skepticism and constructive dialog, not oversimplified contrarian polemics.
Dr. Hall nailed it (as usual). Dr. Spector could have made some very valuable points about the complexity and difficulty of interpreting scientific evidence in medicine or in designing clinical trials. Instead, he decided to go all Mike Adams on us.
Since I’m the one writing this particular post, I won’t bother to quote from my own letter in response to Dr. Spector’s article. If you’re enough of a glutton for punishment or for some odd reason need more David Gorski in much the same way that the world needs more Mark Crislip, feel free to read it yourself. Regardless of your opinion of my characteristic verbosity and false sense of modesty, at the risk of using a reverse argumentum ad populum, I will point out that, in the several months since Dr. Spector’s article appeared, I had a hard time finding any medical skeptic with anything positive to say about it. Still, just because Dr. Spector’s article garnered a lot of negative attention in the skeptical blogosphere doesn’t necessarily mean that he didn’t make some valid points. No doubt, if he is as right as his dogmatic certainty would appear to indicate, Dr. Spector will have prepared excellent rebuttals to our criticisms that address each salient point that each of us raised, at least within the space constraints allowed (which, in all fairness, were significant), right?
Wrong.
Dr. Spector doubles down
The first thing I noticed reading Dr. Spector’s response to criticism is that Dr. Spector does not respond particularly well to criticism. Perhaps he should become a blogger. Of course, not responding well to criticism is not necessarily that horrible a thing in and of itself, as long as it doesn’t devolve into never listening to anyone and some well-intentioned criticism can still get through. On the other hand, these latter two conditions can be a death knell for a scientist, because one of the chief skills needed to succeed in science other than the ability to do good science is the ability to handle criticism with grace and to respond to it with evidence. Outside of science, though, the ability to double down when criticized can be a required positive job skill, particularly in politics, public relations, and, of course, punditry. How else can one explain Rush Limbaugh, Sean Hannity, and Ann Coulter? Like Limbaugh, Hannity, or Coulter, Dr. Spector reacts to criticism primarily by doubling down on his previous statements.
The second thing that I noticed in Dr. Spector’s response is that he is the poster child for a common saying we have in medicine that goes something like, “‘Perfect’ is the enemy of the good.” In other words, yes, in an ideal world it would be lovely if every single medical study had a perfect hypothesis and followed the rigorous methodology of the randomized clinical trial analyzed using exactly the correct statistical analysis. However, many questions don’t easily lend themselves to that approach, and, more importantly, I point out again that there is a difference between hypothesis-generating studies, which are generally smaller, less controlled, and sometimes don’t even have a hypothesis at all (after all, they’re called “hypothesis-generating” for a reason) and final, hypothesis-testing experiments and clinical trials that lead up to FDA approval. Both have their role in science, the former early on in the discovery process, the latter after the discovery process is well under way and approaching an actual, usable treatment that needs to be validated in a phase III randomized clinical trial. Also, such a rigid insistence on methodological perfection above all else smacks of what a favorite blogger of mine (alas, now retired) Revere called “methodolatry.” No doubt Dr. Spector would retort that I say that as though it were a bad thing, to which I would respond that it is a bad thing when taken to an extreme and Dr. Spector does take it to an extreme. Each type of study has its proper place in medical research, from basic science, to translational, to animal studies, to preliminary clinical trials, to the final, big phase III trials whose results are used to apply for FDA approval.
Be that as it may, the last thing that comes through in Dr. Spector’s response is that he appears to think that we do not understand his arguments or the basics of medical science. In fact, Dr. Spector arrogantly doubles down on his previously bad arguments while at the same time accusing skeptics of not understanding him:
It is clear from reading these letters that the correspondents do not understand how to evaluate the medical literature—how to discriminate the wheat from the chaff. They do not understand the hierarchical nature of proffered evidence. For example, I doubt any of the correspondents understand Hill’s criteria for assessing causal relationships in epidemiology/observation studies (see below). I recommend they read Spector and Vesell (2006) or the online methodological supplement to my paper on nutrition in SKEPTICAL INQUIRER (Spector 2009). They should also consult the FDA regulations for what constitutes proof in drug development.
I realize that we’re not supposed to use NSFW language here on SBM. Heck, I’m even the managing editor who says that we bloggers are not supposed to use NSFW language here on SBM, with Dr. Novella’s agreement. So instead of saying what I was thinking as I read this paragraph, I will instead say, “What a pair of fetid dingo’s kidneys!” In any case, I’ll simply point out that reading the medical literature and “separating the wheat from the chaff” is part of my job. It’s what I do, both in my “real world” job and as editor and blogger at SBM. Readers can peruse my work here at SBM and judge for themselves whether I (and all the other bloggers) do a good or poor job at it. As for not understanding “Hill’s criteria,” well, I must admit to a tiny chuckle when I read that sentence, given how often we’ve written about it here on SBM in the context of countering the use and abuse of Hill’s criteria by promoters of unscientific medicine. In fact, Mark Crislip, who wrote one of the letters, also wrote a rather nice discussion of Hill’s criteria. While no one would expect that Dr. Spector would have searched SBM for posts on Hill’s criteria, it is rather presumptuous of him to leap to such conclusions based solely on our letters.
It’s also rather presumptuous of Dr. Spector to assume that I (or others) didn’t read any of his other papers. For instance, I read Spector and Vesell (2006). I also read his SI article Science and Pseudoscience in Adult Nutrition Research and Practice. The latter article spent a great deal of verbiage to conclude that “In general, the most powerful method to establish the truth of many nutritional hypotheses, when contributory causality is postulated, is the prospective, randomized, controlled trial.” No kidding. Nowhere in the article, however, does Dr. Spector acknowledge just how difficult, sometimes impossible even, it often is to do a prospective, randomized, controlled trial studying hypotheses involving the role of nutrition in disease. Sure, it’s possible to do for a subset of nutritional trials (supplementation with individual vitamins or nutrients, for instance), but even then it’s not easy. When it comes to more general questions, such as whether a diet rich in certain fruits and vegetables prevents cancer, it might be either too expensive or too difficult to ensure adequate adherence to control and experimental diets, making such a trial impractical or even downright impossible. What would Dr. Spector have us as scientists do when confronting such questions? Throw up our hands and say, “Since we can’t do a rigorous, randomized, controlled clinical trial, I guess we’ll never know the answer”? Apparently so. I would counter: For such questions, science is either forced to use less rigorous methodology and try to determine if there will be a confluence of evidence from a variety of techniques that leads to an answer or give up, and I’m not one for giving up on important questions. Once again, Dr. Spector doubles down:
Three correspondents claim I don’t understand that the evaluation of clinical science is not easy. Gorski talks about the “messiness of science-based medicine.” Crislip states “medicine advances slowly and erratically.” Hall states that “scientists are doing the best they can.” What they do not understand is that one of the reasons for this sorry state of much of the clinical literature is that many published studies do not have a clear hypothesis, are underpowered, or use inadequate methods (e.g., epidemiology/observation studies attempting to “prove” causal connections).
One wonders what Dr. Spector thinks of the epidemiological studies that first demonstrated a causal link between smoking and lung cancer. It’s notoriously difficult (albeit not impossible), for example, to induce lung cancer in experimental rodents using tobacco smoke or extracts from tobacco smoke. It would also be highly unethical to do a randomized clinical trial to test whether smoking causes lung cancer in humans. Seriously. I ask Dr. Spector in all earnestness how he would have demonstrated that smoking causes lung cancer in humans, if not accepting strong results from epidemiological studies as sufficient.
Ironically, Austin Bradford-Hill (yes, that Hill, of Hill’s criteria) was a pioneer in demonstrating the link between smoking and lung cancer, and his criteria came partially out of that work. Teaming up with Sir Richard Doll, Austin Bradford-Hill published a case-control study in 1950 that showed that smoking greatly increases the risk of lung cancer. The two followed up that study with a long-term prospective study known as the British Doctors Study, which followed the health of 30,000 British doctors for several years, which was a prospective cohort study. Neither of these were controlled, randomized clinical trials. In any case, Hill’s criteria were designed specifically to determine when a disease or condition associated with an environmental exposure could be considered to be caused by that exposure, which, of course, can’t ethically be done in general through a randomized clinical trial. Thus, Dr. Spector implicitly recognizes that for environmental exposures causality can be scientifically inferred from a confluence of epidemiology and other evidence. For everything else, apparently, if it isn’t a randomized clinical trial, to Dr. Spector it’s crap. In any case, Dr. Spector seems to leap back and forth between advocating randomized clinical trials for everything and invoking Hill’s criteria, the latter of which do not necessarily require randomized clinical trials specifically, although #8 (experiment) can, depending on the specific exposure, be interpreted as requiring an RCT of some kind. The problem is, again, that because it’s not acceptable ethically to do an RCT for an environmental exposure that results in disease in a lot of questions, Hill’s criteria #8 will automatically go unfulfilled.
Dr. Spector seems particularly stung over the criticisms of his take on hormone replacement therapy, so much so that he spends a considerable fraction of his limited space in answering them, pointing out that estrogen can cause mammary cancer in female rodents (true) and that it can cause uterine cancer in humans (also true). All of this is rather beside main criticisms I had, given that in general I accept that mixed progestin/estrogen HRT increases the risk of estrogen receptor-positive breast cancer, although more recent evidence suggests that estrogen-only HRT probably does not, or if it does it does so so modestly that it is difficult to detect the increased risk in epidemiological studies. I also can’t help but point out that more recent work suggests that in some cases estrogen HRT can be protective against breast cancer. Once again, the evidence is more complex than how Dr. Spector represents it, as this analysis of the Women’s Health Initiative Estrogen-Alone Trial demonstrates.
In any case, Spector’s arguments miss the point entirely, as he piously proclaims:
In fact, I believe a good physician should use only proven therapies for which the risk/benefit ratio favors the patients. That was never the case with HRT, a known carcinogen. Fortunately, when my sister asked me if she should take HRT for her menopausal symptoms in the mid-nineties, I said no; it’s too risky and its benefits are uncertain.
How condescending of Dr. Spector! As if we believe that a physician should use unproven therapies for which the risk/benefit ratio doesn’t favor the patient! (Look for that sentence to be quote-mined some day.) The point was that it wasn’t so clear at the time and medicine has to use the best available evidence that it has at the time. Even now, the situation is not as clear as Dr. Spector makes it out to be.
Finally, I must admit that Dr. Spector truly disappointed me with his last paragraph, where he doubles down yet again, in essence referring readers to an article many of whose points were easily refuted:
To answer the questions on cancer chemotherapy, I recommend my SKEPTICAL INQUIRER article “The War on Cancer” (Spector 2010). Moreover, the problems with chemotherapy keep surfacing. For example, it was shown that for lung cancer patients, hospice care with minimalist chemotherapy was better than standard chemotherapy in terms of both quality of care and longevity (Temel et al. 2010). The scandals associated with the so-called targeted chemotherapy—such as the ineffective, incredibly expensive, and risky Avastin in breast cancer treatment—escalate (Tucker 2011). Finally, the authors of a recent long, thoughtful article (Smith and Hillner 2011) argue persuasively, as I did, that there is tremendous overuse of chemotherapy. The authors point out that “some oncologists choose chemotherapy in order to maximize their practice income. A system in which one half the profits in oncology [practice] are from drug sales is unsustainable.” Where is the well-being of the patient in this? A thoughtful, empathetic reader will weep.
Actually, I suspect that if Austin Bradford-Hill were alive today, he’d weep at Dr. Spector’s obsessive worship of his nine criteria to the exclusion of all else, but I digress.
When called out on his claims, note how Dr. Spector retreats and revises his argument to something that is more easily defensible. Remember, Dr. Spector didn’t argue just that chemotherapy is overused and that it doesn’t do much good in certain circumstances, such as advanced solid malignancies like non-small-cell lung cancer. He went far, far beyond simply suggesting that we need to be more science- and evidence-based in our use of chemotherapy. In fact, let’s revisit what he did say, which was to label chemotherapy as a “deadly medical hypothesis”:
From a cancer patient population and public health perspective, cancer chemotherapy (chemo) has been a major medical advance.
I also note that I spent quite a bit of verbiage in my last post on this issue refuting much of what Dr. Spector wrote in his “war on cancer” article; so I will not repeat it here. I also pointed out how including chemotherapy as a treatment for cancer as one of his Seven Deadly Medical Hypotheses was off-base and exaggerated the problems with chemotherapy as a cancer treatment far beyond the oncological evidence. On the one hand, while it disappoints me that Dr. Spector apparently never bothered to click on the link I provided to my original criticism of his article, it does not surprise me.
As for the rest, Temel et al is a relatively small but nonetheless provocative study telling us something we already know, namely that chemotherapy doesn’t do very much good for metastatic non–small-cell lung cancer, and I’ve already written quite a bit about the problems with Avastin in breast cancer. This is a controversy that has gone both ways in breast cancer even as Avastin has proven value in other cancers, such as colorectal cancer. Again, if all Dr. Spector had argued was that chemotherapy is overused in some cancers, he’d have gotten little or no argument from me. As for what Smith and Hillner actually argue in their NEJM article, it is not primarily that there is huge overuse of chemotherapy, but rather that oncologist behavior should be changed in the following ways:
     1.  Target surveillance testing or imaging to situations in which a benefit has been shown.
     2.  Limit second-line and third-line treatment for metastatic cancer to sequential monotherapies for most solid tumors.
     3.  Limit chemotherapy to patients with good performance status, with an exception for highly responsive disease.
     4.  Replace the routine use of white-cell–stimulating factors with a reduction in the chemotherapy dose in metastatic solid cancers.
     5.  For patients who are not responding to three consecutive regimens, limit further chemotherapy to clinical trials.
Among other things, they also suggest switching to symptom-directed care when chemotherapy success is highly unlikely. While there is an implication that chemotherapy is overused, the article is more about sticking as closely as possible to science- and evidence-based treatment and recalibrating expectations in order to activate hospice earlier in cases when treatment success can’t be expected. None of these are radical; none of these question the paradigm of chemotherapy use against cancers for which its use is evidence-based. All the authors do is to advocate for integrating hospice care and sticking to evidence-based chemotherapy regimens whose toxicity is lower, all in order to increase the quality of life of cancer patients.
Science: A spectrum of evidence
As I’ve pointed out before, Dr. Spector appears to be someone whose career has been primarily involved in drug development in the later stages. At least, if that’s not the case, I have a hard time figuring out how he can (1) conflate the necessary evidentiary burden required for hypothesis-generating experiments with late-stage randomized clinical trials designed to win FDA approval; (2) lecture critics with a straight face by telling us to read FDA regulations regarding what constitutes proof in drug development and referring us to a paper of his in which he concludes the breathtakingly obvious, namely that the controlled randomized clinical trial is the gold standard for clinical evidence; and (3) dogmatically present a simplified view of scientific and clinical evidence that brooks no ambiguity. All of this bespeaks a narrowness of vision that probably serves him well for defined, limited projects that ask questions like, “How can we design a drug to target this receptor?” and “How do we confirm or refute that the drug works?” However, when it comes to vision that will lead to medical breakthroughs that aren’t simply using existing knowledge to design better new drugs? Not so much, given his obvious contempt for the necessarily less rigorous preliminary studies, the majority of which will fail but a few of which will lead to more rigorous studies demonstrating the efficacy of a new therapy. In other words, sticking to Dr. Spector’s methodology will produce science that is rigorous, safe, and completely uninnovative. I also can’t help but wonder when was the last time Dr. Spector actually took care of a patient, given his obvious intolerance for ambiguity.
No one is arguing against more scientific rigor, least of all me, and where Dr. Spector notes deficiencies in the science being presented that seem reasonable, I’ll be with him. However, when he labels medical hypotheses that are not really hypotheses as deadly when they are not really deadly and then doubles down on the same pseudoskeptical arguments when criticized for them, I’m most definitely not with him there. It needs to be remembered that each of the types of experiments and studies used in science, from small basic science investigations to hypothesis generating genome-wide association studies and next generation sequencing studies of tumors, to the most rigorous phase III clinical trials designed to win FDA approval, are all tools designed to answer specific questions. Each has its place, and a “one size fits all” approach to scientific evidence doesn’t work. Demanding FDA-level evidence for every question makes little sense. This is not a straw man, either; it is what Dr. Spector genuinely appears to be advocating, at least to me. The problem is that Dr. Spector dogmatically clings to the methodolatry of his randomized clinical trials and his FDA regulations in much the same way that a minister, rabbi, or imam holds on to his holy book. That might be a form of science, but it is not good science, and it’s definitely not skepticism, which is why I hope not to see any more of Dr. Spector’s work in SI.

   Evidence in Medicine: Experimental Studies
Steven Novella
Link
Several weeks ago I wrote the first in a brief series of posts discussing the different types of evidence used in medicine. In that post I discussed the role of correlation in determining cause and effect.
In this post I will discuss the basic features of an experimental study, which can sere as a check-list in evaluating the quality of a clinical trial.
Medical studies can be divided into two main categories – pre-clinical or basic science studies, and clinical studies. Basic science studies involve looking at how parts of the biological system work and how they can be manipulated. They typically involve so-called in vitro studies (literally in glass) – using test tubes, petri dishes, genetic sequencers, etc. Or they can involve animal studies.
Clinical trials involve people. They are further divided into two main categories – observational studies and experimental studies. I will be discussing experimental studies in this post – studies in which an intervention is done to study subjects. Observational studies, on the other hand, look at what is happening or what has happened in the world, but does not involve any intervention.
Experimental Studies
The primary advantage of experimental studies is that they allow for the direct control of variables – in the hopes of isolating the variable of interest. Results are therefore capable of being highly reliable, although good clinical experiments are difficult to design and execute. When assessing a clinical trial here are the features to examine.
Prospective vs Retrospective
A prospective trial is one in which the treatment and the outcomes are determined prior to any intervention being done. Experimental trials are almost by definition prospective. A retrospective trial is one in which the data is gathered after the fact – taking patient records, for example, and looking at treatments and outcomes.
In a retrospective study you can try to account for variables, but you cannot control for them. It is therefore much more likely that there are confounding factors and the results are not as reliable.Also, retrospective studies can be biased by the way information is obtained – there can be a bias in the way patients are identified, for example.
Prospective trials are therefore considered superior to retrospective trials, which are at best preliminary in their conclusions.
Placebo-Controlled
Not all prospective trials are placebo-controlled, however. A non-controlled trial might identify potential subjects, give them all a treatment, and then see how they do. Such open-label single arm trials cannot control for placebo effects or experimenter biases, and again results should be considered preliminary.
Open or uncontrolled trials are not useless, however. The outcome of subjects in such trials can be compared to historical controls, and if a significant result is apparent (along with safety) can be used to justify a larger and more rigorous trial.
Controlled trials have one or more comparison groups in the trial itself – different groups of subjects receive different treatments or no treatment. All subjects can be followed in same manner. Control groups allow the experimenter to make sure that all the subjects have the same disease or symptoms, that they receive known treatments, and many variables (such as other treatments they may be receiving, severity at inclusion, age, sex, race, etc.) can be accounted for.
Controlling for variables
With controlled trials the experimenter can start to control for variables. If the question is – does treatment A improve outcome in disease X, a controlled prospective trial can attempt to isolate treatment A from other factors that may affect outcome.
One method for controlling variables is stratification – the study protocol can place subjects in different treatment groups so that the groups end up with the same proportion of different sexes, ages, races, and other known variables that may be pertinent. Stratification can control for known or obvious confounding factors.
But of course there can always be unknown confounding factors. The only way to deal with these is through randomization and large study size. If a large number of subjects are randomly assigned (once stratified for age, sex, etc.) into the different treatment groups, then any unknown variables should average out. Of course, this requires sufficient numbers – small studies are always suspect because the groups may be significantly different by chance alone.
Randomization is important because when patients select their own treatments this opens the door for selection bias. For example, sicker patients may opt for more aggressive therapy. They will do worse because they were sicker to begin with, making the more aggressive therapy look less effective.
Blinding
A randomized prospective trial can control for many variables, but the only way to control for placebo effects and the bias of the experimenters is with blinding – meaning that participants don’t know who is getting the real treatment and who is getting a different treatment or a placebo.
A single-blind study is one in which subjects do not know which treatment they are getting. A double-blind study is one in which the experimenter does not know either – until the study is done and the “code is broken.”
When subjects are blinded, placebo effects should be the same. It is often difficult, however, to fully blind subjects. Medications may have obvious side effects, and subjects who experience the side effects know they are getting active medication.
Physical interventions, like acupuncture, surgery, massage, or physical therapy, are difficult to impossible to blind. A person knows if they have been massaged or not. For these studies creative blinding techniques may need to be used. Or, “sham” procedures can be used for placebos.
Studies may also assess how successful the blinding was – by asking subject if they think they received the placebo or the treatment.
Experimenters also need to be blinded to eliminate placebo and biasing effects. This is easy for drug trials, but may be impossible for physical intervention trials. However, a study can be partially double-blinded if there is a blinded evaluator – an experimenter whose only involvement with the study is to assess the subjects, while carefully avoiding any information that would clue them in as to which treatment arm each subject was in.
But the best studies are ones in which everyone involved is completely blinded until the results are completely in.
Outcome measures
Deciding how to determine if an intervention “works” is not always trivial. Outcome measure need to be a good and reliable marker of the disease or syndrome you are following. For example, in a diabetes study, do you follow HgA1C, random glucose checks, glucose tolerance tests, end-organ damage, need for medication, or some other biological marker?
In addition to being a good marker for what you are studying, the outcome should be meaningful. Do we care if a cholesterol lowering drug lowers total cholesterol, or if it prevents heart attacks and strokes? And if it prevents heart events, does it prolong survival (or just reduces angina without affecting survival)?
Outcomes also need to be free of confounding. For example, early stroke trials looked at stroke incidence, which may seem reasonable. However, if more subjects on a treatment died of heart attacks, they would not be around to have a stroke, so the treatment reduces stroke but only by allowing more heart attacks. So stroke-free survival is a better outcome to follow.
Outcome measures also vary on how objective or subjective they are. Just asking patients how they feel is not a very reliable outcome measure. You can pseudo-quantify this by asking them to put a number on their pain or other symptoms, but it is still a subjective reports. Measuring the volume of lesions in the brain, however, is an objective outcome measure, and is therefore more reliable.
Many studies will follow several outcomes – some subjective but important, and others objective and quantifiable if an indirect marker rather than a direct outcome we care about.
Statistical analysis
I won’t go into statistics in any detail, as that is a highly technical area and any reasonable treatment would be much longer than the rest of this post. Here even medical professionals rely upon statistical experts to make sure we get it right.
But it is good to understand the basics (as long as you don’t rely upon basic knowledge – then it is easy to be fooled by fancy statistical tricks).
The most basic concept of clinical trials is statistical significance – is there an effect or correlation that is probably greater than chance. Most studies rely upon the P-value, which is a measure of the chance the result occurring if the the null hypothesis (no effect) is correct. A P-value of 0.05 means (roughly) that 5% (or 1 in 20) probability that the outcome is due to chance alone, and not a real effect. P-value of 0.05 is commonly used as a cutoff of statistical significance, but it is important to realize with this cutoff 1 in 20 studies of worthless treatments will appear positive due to chance alone. Lower P-values, such as 0.01, are more significant.
But P-value isn’t everything. A poorly designed study can result in an impressive P-value. Also, the size of the effect must be considered. You can have a low P-value for a tiny effect (if there are large numbers of subjects in the trial) – the effect may be clinically insignifcant, and small effects are more likely to be due to hidden biases or confounders.
Therefore, we generally are only impressed when a clinically large effect also has a low P-value.
In addition to P-value, the number of subjects in the trial is very important. Even though these are related, the larger the study the more impressive the results, as random fluctuations are less likely to play a role.
One common trick to look out for is multiple analysis. A study may, for example, look at 10 variables (or one variable at 10 different points in time), and find statistical significance for one, and present that as a positive study. However, this is equivalent to taking 10 chances at that 1 in 20 chance of hitting significance. Proper statistical analysis will account for multiple comparisons.
Other factors to look out for
There are features that are important to consider is evaluating a clinical trial. What was the dropout rate? If half of the subjects dropped out, that unrandomizes or biases the groups, because drop outs are not random. For example, subjects that do not respond to treatment may drop out, leaving only those who do well.
Not all controls are equal as well. Sometime the control group is not an inactive placebo but standard care. What if the standard treatment is too effective, or what if it is not effective at all. You need to know what the study treatment is being compared to.
Conclusion
When a new clinical trial is being promoted in the news as evidence for or against a treatment – run down this list. Is it a randomized, controlled, double-blind trial, is the blinding adequate, are the outcome measures objective and relevant, is the effect size robust, how large is the study, what variables are actually being isolated, and what was the drop out rate?
And of course, no one study is ever the definitive last word on a clinical question. Each study must be put in the context of the full scientific literature, which means considering plausibility or prior probability. That is the essence of science-based medicine.

   Evidence in Medicine: Correlation and Causation
Steven Novella
Link
There are two general approaches to subverting science-based medicine (SBM): anti-science and pseudoscience. Anti-scientific approaches are any that seek to undermine science as the determinant of the standard of care, often overtly advocating for spiritual or subjectively-based standards. Some attack the validity of science itself, usually with post-modernist philosophy.
Pseudoscientific proponents, on the other hand, praise science, they just do it wrong. In reality there is a continuum along a spectrum from complete pseudoscience to pristine science, and no clear demarcation in the middle. Individual studies vary along this spectrum as well – there are different kinds of evidence, each with its own strengths and weaknesses, and there are no perfect studies. Further, when evaluating any question in medicine, the literature (the totality of all those individual studies) rarely points uniformly to a single answer.
These multiple overlapping continua of scientific quality create the potential to make just about any claim seem scientific simply by how the evidence is interpreted. Also, even a modest bias can lead to emphasizing certain pieces of evidence over others, leading to conclusions which seem scientific but are unreliable. Also, proponents can easily begin with a desired conclusion, and then back fill the evidence to suit their needs (rather than allowing the evidence to lead them to a conclusion).
For example, the anti-vaccine movement systematically endorses any piece of evidence that seems to support the conclusion that there is some correlation between vaccines and neurological injury. Meanwhile, they find ways to dismiss any evidence which fails to show such a connection. They, of course, accuse the scientific community of doing the same thing, and each side cites biases and conflicts in the other to explain the discrepancy. It is no wonder the public is confused.
How, then, do we use the evidence to arrive at reliable scientific conclusions? That is what I will be discussing in this series of posts, beginning with a discussion of correlation and causation, but here is a quick overview: SBM is achieved through a consideration of scientific plausibility and a systematic review of the clinical evidence. In other words – all scientific evidence is considered in a fair and thorough manner, including basic science and clinical evidence, and placed in the context of what we know about how the world works. Further, we do not rely upon any individual’s systematic review of the evidence, but on the consensus of analysis among experts and institutions – so that any biases are likely to average out.
This leads us to the final continuum – the consensus of expert opinion based upon systematic reviews can either result in a solid and confident unanimous opinion, a reliable opinion with serious minority objections, a genuine controversy with no objective resolution, or simply the conclusion that we currently lack sufficient evidence and do not know the answer. It can also lead, of course, to a solid consensus of expert opinion combined with a fake controversy manufactured by a group driven by ideology or greed and not science. The tobacco industry’s campaign of doubt against the conclusion that smoking is a risk factor for lung cancer is one example. The anti-vaccine movement’s fear-mongering about vaccines and autism is another.
Correlation and Causation
Much of scientific evidence is based upon a correlation of variables – they tend to occur together. Scientists are careful to point out that correlation does not necessarily mean causation. The assumption that A causes B simply because A correlates with B is a logical fallacy – it is not a legitimate form of argument. However, sometimes people commit the opposite fallacy – dismissing correlation entirely, as if it does not imply causation. This would dismiss a large swath of important scientific evidence.
For example, the tobacco industry abused this fallacy to argue that simply because smoking correlates with lung cancer that does not mean that smoking causes lung cancer. The simple correlation is not enough to arrive at a conclusion of causation, but multiple correlations all triangulating on the conclusion that smoking causes lung cancer, combined with biological plausibility, does.
Correlation must always be put into perspective. There are two basic kinds of clinical scientific studies that may provide evidence of correlation – observational and experimental. Experimental studies are ones in which some intervention is given to a study population. In experimental studies it is possible to control for many variables, and even reasonably isolate the variable of interest, and so correlation is a well-designed experimental study is very powerful, and we generally can assume cause and effect. If active treatment vs placebo correlates with a better outcome, then we interpret that as the treatment causing the improved outcome. (“Well-designed” is the key here – a subject of a future post.)
In observational studies populations are observed in the real world, but no intervention is being given. Observational studies can be very powerful, because they can look as extremely large numbers of subjects (more than is practical in an experimental study) but the weakness is that all variables cannot be controlled for. Researchers can account for known variables (race, age, and sex are common), but it is always the unknown variables that can confound such studies.
In observational studies lack of correlation is easier to interpret than a positive correlation – if there is no correlation between A and B then we can pretty much rule out a causal relationship. The only caveat is that a correlation is being obscured by a factor that was not accounted for. When a correlation is found in observational studies – that is when the assumption of cause and effect must be avoided, and more thorough analysis is required.
If A correlates with B, then A may cause B, B may cause A, A and B may be caused by a common variable C, or the correlation may be a statistical fluke and not “real”. Further studies are then required to confirm the correlation and any specific causal hypothesis.
To use the smoking example again – the hypothesis that smoking causes cancer as the causal relationship to explain the correlation raises several predictions, all later confirmed. The duration of smoking increases risk of cancer (a dose response relationship), stopping smoking reduces the risk of cancer, greater intensity of smoking increases risk, and smoking unfiltered vs filtered cigarettes is associated with higher risk. These various correlations only make sense if smoking causes lung cancer. Further, tobacco smoke contains substances demonstrated to be carcinogens – so there is biological plausibility.
The greatest abuse of the correlation equals causation fallacy is the assumption of cause and effect from a single anecdotal case. Here we are not talking about an observational study where statistics are brought to bear on hundreds or thousands of subjects, but the uncontrolled observation of a single individual. Such cases are very compelling to the human psyche- we are more moved by stories than statistics. But they make for very weak scientific evidence. This is not to say they are worthless – even a single case can raise the question of a possible correlation. But they cannot be used to establish even that a correlation is real. (Anecdotes generate questions, not answers.)
Again to use the anti-vaccine movement as an example, it is easy to generate fear based upon individual cases of bad outcomes after receiving the vaccine. We are hard-wired to find such events compelling. But such correlations should and do occur on a regular basis, even without any causal factor. Further, it is natural, after a new disease or disorder appears, to think back over any recent events that may explain it. Our minds will latch onto anything that sticks out, and over time our memories will even morph to make the apparent correlation more compelling.
Here is an anti-vaccine and conspiracy site that is essentially collecting anecdotes of correlation between the flu vaccine and miscarriages among women. This goes beyond the assumption of cause and effect from correlation, to the assumption of correlation from anecdote. Common things occur together commonly. Given the number of spontaneous miscarriages, and the number of pregnant women receiving the flu vaccine, we would expect there to be thousands of women who miscarry within 24 hours of receiving the flu vaccine, just be chance alone. So first we have to ask – is this a real correlation?
The answer, according to systematic reviews of existing evidence, is no. There is no apparent risk of adverse outcome in pregnancy from the flu vaccine.
The media tends to develop a narrative they think will sell, and then that becomes the story they are telling. In the midst of this severe flu season, the narrative the media is telling is of dramatic adverse events following the flu vaccine. These events are always there, because people are always getting sick, and when you vaccinate millions of people, some of them will get sick afterwards by chance alone. Individual stories are therefore misleading – we need statistics on large number of people to arrive at any conclusions. But statistics don’t make headlines – individual stories do.
The CDC and the WHO are tracking the statistics as the seasonal and H1N1 flu vaccines are rolled out. In the end, these statistics will tell the story.
In conclusion, correlation is an extremely valuable type of scientific evidence in medicine. But first correlations must be confirmed as real, and then every possible causational relationship must be systematically explored. In the end correlation can be used as powerful evidence for a cause and effect relationship between a treatment and benefit, or a risk factor and a disease. But it is also one of the most abused types of evidence, because it is easy and even tempting to come to premature conclusions based upon the preliminary appearance of a correlation.

Why Do We Really Need Clinical Trials?
Brennen McKenzie on January 6, 2012
Link
A point I make over and over again when talking about new or alternative therapies that are not supported by good clinical trial evidence is that lower-level evidence, such as theoretical justifications, anecdotes, and pre-clinical research like in vitro studies and animal model testing, can only be suggestive, never reliable proof of safety or efficacy. It is necessary to begin evaluating a new therapy that does not yet have clinical evidence to support it by showing a plausible theory for why it might work and then moving on to demonstrate that it actually could work through pre-clinical research, which includes biochemistry, cell culture, and animal models. These sorts of supporting preclinical evidence are what we refer to when we refer to the “prior plausibility” of a clinical study. But this kind of evidence alone is not sufficient to support using the therapy in real patients except under experimental conditions, or when the urgency to intervene is great enough to balance the significant uncertainty about the effects of the intervention.
In support of this conclusion, we can consider the inherent unreliability of individual human judgments and all the many ways in which inadequately controlled research can mislead us. And we can reflect on how promising results in early trials often melt away when better, larger, more rigorous studies are done that better control for bias (the so-called Decline Effect). And it is not at all difficult to compile a large list of examples of the harm inadequately studied medical interventions can cause.
But what I’d like to do here is focus on a particularly good specific example of why thorough clinical trial evaluation of promising ideas is not just a nice extra to confirm what we already believe is true, it is the only way to genuinely know whether our treatments to more good than harm.
SELECT: What Is It About?
The Selenium and Vitamin E Cancer Prevention Trial (SELECT) was initiated because, according to the National Cancer Institute (NCI) at the National Institutes of Health (NIH), “Evidence from epidemiologic studies, observational studies, and clinical trials looking at preventing cancers other than prostate cancer had suggested that selenium and/or Vitamin E might prevent prostate cancer.” The NCI describes some of the specific studies that suggested selenium and Vitamin E supplementation might protect against prostate cancer.
Selenium
Selenium is a nonmetallic trace element found in food, especially plant foods such as rice and wheat, seafood, meat, and Brazil nuts. Selenium is an antioxidant and may help control cell damage that could lead to cancer.
The Nutritional Prevention of Cancer (NPC) study, first reported in 1996, included 1,312 men and women who had a history of non-melanoma skin cancer. Results of the trial showed that men who took selenium to prevent new non-melanoma skin cancers received no benefit from selenium in preventing that disease. However, approximately 60 percent fewer new cases of prostate cancer were observed among men who had taken selenium for six and one-half years than among men who took placebo (1). In a 2002 follow-up report, the data showed that men who took selenium for more than seven and one-half years had about 52 percent fewer new cases of prostate cancer than men who took placebo (2). This trial was one of the reasons for studying selenium in SELECT.
Vitamin E
Vitamin E is in a wide range of foods, especially vegetables, vegetable oils, nuts, and egg yolks. Vitamin E, like selenium, is an antioxidant, which may help control cell damage that can lead to cancer.
In the 1998 study of 29,133 male cigarette smokers in Finland (known as the Alpha-Tocopherol, Beta-Carotene Trial, or ATBC), 32 percent fewer new cases of prostate cancer and 40 percent fewer deaths from prostate cancer were observed among men who took Vitamin E in the form of alpha-tocopherol to prevent lung cancer than among men who took a placebo.
So in 2001, the trial began enrolling subjects in a randomized, blinded, placebo-controlled prospective clinical trial to see if supplementation of selenium and Vitamin E, or a combination of the two, would reduce the incidence of prostate cancer. By 2004, over 35,000 men in the United States, Canada, and Puerto Rico had been enrolled. The subjects were randomly assigned to receive:
    • Selenium and Vitamin E
    • Selenium and a placebo
    • Vitamin E and a placebo, or
    • Two placebos. Two placebos were used in the trial: one looked like a selenium capsule; the other looked like a Vitamin E capsule. Each placebo contained only inactive ingredients. Neither the participants nor the researchers knew who received the selenium and Vitamin E, or the placebos, a process known as blinding or masking.
The trial was originally planned for between 7 and 12 years of supplementation and then subsequent followup to monitor for the development of prostate cancer, and it was designed to detect a 25% reduction in the incidence of prostate cancer, which was felt to be an achievable and clinically meaningful difference.
So How Did It Work Out?
In 2008 the investigators reviewed the data up to that point and found no sign of a protective effect from selenium or Vitamin E, separately or taken together. It was judged unlikely that the 25% reduction in risk that was the criteria for effect in the study would be seen even if the study continued, so the supplementation portion of the study as halted. The subjects were told which supplement or placebo they had been given and were instructed to stop taking their supplements.
At that time, a greater number of cases of prostate cancer had developed in the group taking only Vitamin E than in the other groups. However, the difference did not reach statistical significance, so the investigators could not say whether it was a meaningful difference or only the result of chance.
Subsequently, the men have continued to be monitored, and the difference in the rate of prostate cancer between men who had been taking only Vitamin E supplements and the other men in the study has continued to grow even after they stopped taking the supplement. The difference is now statistically significant, so it likely represents a real biological phenomenon, not simply random chance. Men who were in the Vitamin E alone group have developed prostate cancer at a rate 17% higher than the men in the other groups.
Those men who took selenium alone or selenium and Vitamin E also have a higher rate of prostate cancer than those men who received the placebos, but this difference is not yet statistically significant. No other differences suggesting a beneficial or harmful effect of either supplement have been detected.
So Why Didn’t It Work Out As Expected?
From a biological point of view, the reason why Vitamin E supplementation turned out to be a risk factor for development or prostate cancer instead of a protective factor aren’t yet known. Nor is it clear why a statistically significant increase in risk hasn’t yet been found for those men who took Vitamin E and selenium together. There are, of course, theories as to how precisely Vitamin E influences prostate cancer risk, and the NCI has solicited research proposals from other investigators to use the data and samples collected in SELECT to investigate this question.
But from the point of view of epistemology, how we figure out what works and what doesn’t, there are a number of reasons why the finding shouldn’t be entirely surprising. For one thing, new ideas often turn out to be wrong. The degree of plausibility to the underlying theory and the amount of supportive pre-clinical research evidence are useful in determining roughly how promising an idea is, but ultimately the proof is in the pudding, with the pudding being multiple replicated independent high-quality randomized clinical trials. Given the incredible number of complex, interacting factors involved in the development of each particular disease, it shouldn’t surprise us when even a reasonable hypothesis with limited support from observational trials and other lower level evidence turns out to be false.
And in the case of this particular hypothesis, the preliminary evidence which first suggested the need for the SELECT in the first place didn’t work out so well either. As already mentions, “results of the [Nutritional Prevention of Cancer] trial showed that men who took selenium to prevent new non-melanoma skin cancers received no benefit from selenium in preventing that disease.” And there has since been some indication of potential harm, though the data is conflicting; “Since the start of SELECT, four studies have been published on the effect of selenium on blood glucose and risk of diabetes. Two studies suggested that higher levels of selenium taken from supplements or received naturally were associated with an increased risk of diabetes. One study showed no such association, and one showed that people with higher levels of selenium in their blood had a reduced risk of diabetes (3-6).”
As for Vitamin E, the NCI states, “There are no clinical trials that show a benefit from taking vitamin E to reduce the risk of prostate cancer or any other cancer or heart disease (7, 8, 9-13).” And the subjects in the HYPERLINK “http://www.cancer.gov/newscenter/qa/2003/atbcfollowupqa” ATBC trial who took Vitamin E had a 50% increase in the occurrence of hemorrhagic strokes, an 18% increase in the occurrence of lung cancer, and an 8% overall greater mortality than control subjects. So not only were the initial promising results suggesting a benefit for prostate cancer not correct, there were fewer benefits and greater risks than expected in terms of other diseases as well.
The Bottom Line
The moral of this story is simple and shouldn’t be controversial: While a plausible theory and suggestive preclinical evidence may suggest a new therapy could be beneficial, only multiple, rigorous, high-quality clinical trials can determine if this is actually true. Good ideas turn out to be wrong often enough that our initial enthusiasm with the ideas themselves and with supportive preliminary research data should always be tempered by an understanding that these lower levels of evidence are inherently less reliable than the slow, expensive, complicated process of rigorous clinical studies. And bad ideas, those without a plausible theoretical foundation or supportive pre-clinical evidence, are even less likely to bear fruit when properly examined.
Of course, such high-quality studies may not always be possible. As a veterinarian, I can never expect to have a 7-year prospective study of 35,000 subjects to rely on. So the temptation to act on lesser evidence is understandable. But therapies used without higher-quality evidence to support them are inevitably going to be more likely to fail or to cause harm than therapies that have been more thoroughly studied. We must accept this and factor that knowledge into our clinical decisions. If the need to intervene is great enough, it can be justifiable to employ therapies without high-quality clinical trial evidence to support them. However, we must convey the real degree of uncertainty to our patients or clients, and we must be especially vigilant and honest about potential failures or unintended consequences.
It is common to see sweeping and confident claims of benefit and safety for alternative therapies that, when examined, turn out to be based on suggestive evidence often much weaker than that which led the NCI to study the effect of selenium and Vitamin E on prostate cancer risk. CAM advocates have gotten quite good at throwing up publication references to support their claims, and the hassle factor involved in examining those papers in detail to see what level of evidence they actually provide can discourage anyone from thoroughly investigating these claims. This, of course, leaves the impression that there is good evidence to support what CAM advocates are saying, not only in the minds of the general public but in the minds of physicians, veterinarians, nurses, and others who should know better.
I think much of the success of the “integrative medicine” meme has been based on the lack of an adequate understanding among health professionals about the serious limitations of low-level evidence. The SELECT illustrates nicely how even a plausible intervention with enough low-level evidence to justify a major clinical trial can prove not only less helpful than originally hoped but even actively harmful. The same principle applies to an even greater degree to less plausible hypotheses. High-quality clinical trials are not simply icing on the cake confirming what we already know, they are the cake without which we know a lot less than we usually think.
References
     1.  Clark LC, Combs GF Jr., Turnbull BW, et al. HYPERLINK “http://www.ncbi.nlm.nih.gov/pubmed/8971064” Effects of selenium supplementation for cancer prevention in patients with carcinoma of the skin. A randomized controlled trial: Nutritional Prevention of Cancer Study Group. JAMA 1996; 276(24):1957-1963
     2.  Duffield-Lillico AJ, Reid ME, Turnbull BW, et al. HYPERLINK “http://www.ncbi.nlm.nih.gov/pubmed/12101110” Baseline characteristics and the effect of selenium supplementation on cancer incidence in a randomized clinical trial: A summary report of the Nutritional Prevention of Cancer Trial. Cancer Epidemiology, Biomarkers & Prevention 2002; 11(7):630-639.
     3.  Stranges et al. HYPERLINK “http://www.annals.org/content/147/4/217” Effects of Long-Term Use of Selenium Supplements on the Incidence of Type 2 Diabetes. Annals of Internal Medicine; 147:217-233, 2007.
     4.  Bleys J et al. HYPERLINK “http://www.ncbi.nlm.nih.gov/pubmed/17392543” Serum Selenium and Diabetes in U.S. Adults. Diabetes Care; 30:829-834, 2007.
     5.  Rajpathak et al. HYPERLINK “http://www.ncbi.nlm.nih.gov/pubmed/16093402” Toenail Selenium and Cardiovascular Disease in Men with Diabetes. Journal of the American College of Nutrition; 24: 250-256, 2005.
     6.  Czernichowet et al. HYPERLINK “http://www.ajcn.org/content/84/2/395.short” Antioxidant supplementation does not affect fasting plasma glucose in the Supplementation with Antioxidant Vitamins and Minerals (SU.VI.MAX) study in France: association with dietary intake and plasma concentrations. American Journal of Clinical Nutrition; 84:395-9, 2006.
     7.  Lippman SM, Klein EA, Goodman PJ, et al. Effect of selenium and vitamin E on risk of prostate cancer and other cancers. JAMA 2009; 301(1). Published online December 9, 2008. Print edition January 2009.
     8.  EA Klein, IM Thompson, CM Tangen, et al. Vitamin E and the Risk of Prostate Cancer: Results of The Selenium and Vitamin E Cancer Prevention Trial (SELECT). JAMA 2011; 306(14) 1549-1556.
     9.  Yusuf S, Dagenais G, Pogue J, et al. Vitamin E supplementation and cardiovascular events in high risk patients. The Heart Outcomes Prevention Evaluation Study Investigators. New England Journal of Medicine. 2000;342:154-60.
     10.  Sesso HD, Buring JE, Christen WG, et al. Vitamins E and C in the prevention of cardiovascular disease in men: the Physicians’ Health Study II randomized controlled trial. JAMA. 2008; 300(18):2123-33.
     11.  Lee IM, Cook NR, Gaziano JM, et al. Vitamin E in the primary prevention of cardiovascular disease and cancer: the Women’s Health Study: a randomized controlled trial. JAMA. 2005; 294(1):56-65.
     12.  Lonn E, Bosch J, Yusuf S, et al. Effects of long-term vitamin E supplementation on cardiovascular events and cancer: A randomized controlled trial. JAMA 2005; 293(11):1338-1347.
     13.  Miller ER III, Pastor-Barriuso R, Dalal D, et al. Meta-analysis: High-dosage vitamin E supplementation may increase all-cause mortality. Annals of Internal Medicine 2005; 142(1):37-46.

Wither the randomized controlled clinical trial?
With the rise of precision medicine and genomics, the conventional randomized clinical trial appears more and more outdated. Fortunately, clinical trials are evolving, but will it be enough to incorporate the numerous advances in “-omic” medicine in a rigorous scientific manner to benefit patients?
David Gorski on June 20, 2016
Link

One of the most frequent complaints about evidence-based medicine (EBM), in contrast to science-based medicine (SBM), is its elevation of the randomized clinical trial as the be-all and end-all for clinical evidence for an intervention for a particular disease or condition. Unknown but enormous quantities of “digital ink” have been spilled explaining this distinction right here on this blog, beginning with our founder’s very first post, continuing with Kimball Atwood’s series of posts explaining the shortcomings of EBM’s reliance on clinical trials über alles using homeopathy as his example, to my referring to this aspect of EBM as “methodolatry,” defined as profane worship of the randomized controlled clinical trial (RCT) as the only valid method of clinical investigation. The problem, of course, with methodolatry, is that it completely ignores prior plausibility, and when that prior plausibility is as close to zero as you can imagine (e.g., for clinical trials of homeopathy), then the only positive results that you see in such trials can reasonably be concluded to be due to noise, shortcomings in trial design, and bias. Unfortunately, a failure to realize this has led to many pointless clinical trials and contributed to the rise of a whole new “specialty” known as integrative medicine, dedicated to “integrating” quackery and pseudoscience into science-based medicine.
So we know that practitioners of “complementary and alternative medicine” (CAM), now referred to more frequently as integrative medicine, don’t like RCTs. They love pragmatic trials, because such trials are usually unblinded, often not randomized, and generally face a lower bar of evidence. That pragmatic trials are intended to test the “real world” use of medical and surgical interventions that have already been shown to be safe and effective in RCTs and that the vast majority of CAM nostrums have not met that standard appears not to concern them in the least. However, CAM practitioners are not the only ones critical of RCTs, as I learned when, via Steve Novella, I came across an article in The New England Journal of Medicine entitled “Assessing the Gold Standard — Lessons from the History of RCTs” by Bothwell et al. Given that the article is two weeks old, I wonder how I missed it. Be that as it may, although Bothwell et al make some good points, I tend to agree with Steve that the overall gist of the article is overly critical, to the point of, as Steve put it, portraying the RCT as broken rather than flawed and advocating revolution rather than reform.
The history of RCTs
In 2016, the RCT is so ubiquitous and unquestioned that we frequently forget that what we now know as the modern RCT didn’t exist until the 1940s, when British epidemiologist Austin Bradford Hill formalized RCT methods. Even more unbelievable to someone trained since then, the FDA did not actually require RCTs before drug approval until 1970, which was actually eight years after the Kefauver–Harris Amendments to the Food, Drug, and Cosmetic Act passed in the wake of the thalidomide disaster. That was when the FDA finally interpreted the clause in the Kefauver–Harris Amendments mandating that new drugs be proven safe and efficacious in “adequate and well-controlled investigations” to mean requiring RCTs before the approval of new pharmaceuticals. By the 1980s (which, coincidentally, encompassed the eight years when I was in college and medical school), Bothwell et al note that the RCT had been declared the “gold standard” of medical knowledge. The other relevant history that Bothwell et al recount is that by the 1990s, pharmaceutical companies had surpassed medical academia and the government as the primary producer of RCTs.
To this, I would add that the concept of “evidence-based medicine” wasn’t really formalized until the 1990s, either, complete with its now familiar hierarchies of evidence (with meta-analyses of RCTs and large RCTs themselves at the very top of the hierarchy), making EBM as a construct only a couple of decades old. That’s not to say that medicine hadn’t been evidence- and science-based before that. After all, the Flexner Report, published in 1910, was pivotal in making medicine much more scientific than it had been. It is, however, always humbling to remind oneself that for most of its history medicine hasn’t really been science-based.
I myself have discussed some of these issues, as have others. Mostly, we discuss the history of how RCTs, after being declared the “gold standard” of medical knowledge, became a tool with which advocates of unscientific medicine could ignore the incredibly low prior plausibility of a treatment and go straight to human studies based on a questionable rationale that “people are using them anyway.” As we’ve explained many times, the “blind spot” of EBM is that it relegates basic science knowledge, especially basic science knowledge that shows a particular treatment (e.g., homeopathy or reiki) to be so close to impossible as to be, for all intents and purposes, impossible, to the lowest level on its hierarchy of evidence. The whole idea of RCTs depends on the assumption that the investigation of a medical intervention won’t reach the level of RCTs without considerable preclinical scientific evidence supporting its plausibility.
CAM practitioners aren’t alone, however, in not liking RCTs or finding them too constraining. Bothwell et al note that many clinicians were not exactly thrilled with RCTs when they were first introduced, in particular the possibility of withholding promising new interventions from control groups. Bothwell et al basically provide a rundown of some of these objections c. 2016.
The problem(s) with RCTs: They’re a religion!
My issues with this article don’t mean that it doesn’t make some valid observations, some of which I myself have made. They also concede that RCTs minimize bias and have strengthened the scientific rigor of clinical study. They’ve also been important for identifying some interventions that don’t work, such as the classic example of internal mammary ligation, an operation commonly performed for angina from the 1930s to 1950s based on the rationale that ligating this artery would redirect more blood flow to the coronary arteries. It was an operation that surgeons widely believed to be effective because patients usually demonstrated an impressive improvement in symptoms. Unfortunately, when an RCT, complete with “sham surgery” was carried out, investigators discovered that the patients undergoing sham surgery exhibited the same improvement in symptoms. We’ve since learned that surgery itself induces powerful placebo effects, explaining the previous apparent efficacy of the operation.
Unfortunately, Bothwell et al then devolve into a “special flower” sort of argument against RCTs of surgical interventions:
As more surgical RCTs appeared in the 1960s and 1970s, however, surgeons increasingly recognized their limitations: each patient had unique pathological findings, each surgeon had different skills, and each operation involved countless choices about anesthesia, premedication, surgical approach, instrumentation, and postoperative care, all of which defied the standardization that clinical trials required. Sham controls could not be used for major operations, which limited opportunities for blinded trials.
Such concerns played out in debates about RCTs for coronary-artery bypass grafting (CABG). When the first major RCT of CABG revealed that most patients with chronic stable angina received no survival benefit from CABG critics pounced: the participants were too healthy, the surgeons too inexperienced, the operative mortality too high, and the statistical analysis suspect. Prominent surgeons argued that RCTs were inappropriate for surgery. René Favaloro, who had played a key role in developing CABG, argued that “randomized trials have developed such high scientific stature and acceptance that they are accorded an almost religious sanctification. . . . If relied on exclusively they may be dangerous.”
I can’t help but mention that, to my shame, surgeons not infrequently dismiss the results of RCTs using rationales that the procedure is just too complicated and has too many different moving parts, an argument not unlike that of CAM advocates, who basically argue that RCTs can’t study their woo. Be that as it may, whenever anyone compares anything in science to a religion, my skeptical antennae start twitching fiercely. But, wait, you say. Isn’t my use of the term “methodolatry” exactly the same thing? Not exactly. My use of the term is clearly meant as ironic exaggeration, something most readers will immediately recognize, whereas the surgeon quoted, René Favaloro, was clearly dead serious when he compared RCTs to religion and Bothwell quotes him that way to bolster the criticisms of RCTs. If there’s one thing I’ve observed over the years, it’s how funny it is that you almost never hear anyone criticizing a positive result of an RCT by unironically referring to belief in RCTs as religious. In contrast, it’s not infrequent to hear such a charge made about negative RCTs, usually by members of the specialty that profits from the intervention found wanting. It is depressing that Bothwell et al fell into the trap of uncritically citing objections of the nature those made by Dr. Favoloro.
Of course, as Dr. Prasad notes, it’s impossible to completely prove a negative with a clinical trial, and his retort was so good that I’ll quote it directly:
This strategy of saying negative data is not good enough is akin to asking someone to prove Santa Claus does not exist. After the person does a census of all people in the world, and finds no Santa Claus, believers note that he didn’t look under the ocean, in all of the deserts, etc. You cannot prove an intervention does not work under ANY circumstances, you must instead show that it can work under SOME circumstances. Proponents of stable coronary stenting have FAILED to do that, and now are sabotaging the ongoing ISCHEMIA trial from enrolling enough patients. After all, ISCHEMIA can only erode their 10-15 billion dollar market share.
Does this sound familiar? It’s the same thing CAM advocates do after a negative clinical trial of whatever woo they have tested. They slice and dice the data looking for a subgroup, no matter how small, in which the treatment works, and sometimes the fact that the positive finding was only in a subgroup gets lost in how the message is communicated. My favorite example of this is TACT (Trial to Assess Chelation Therapy), a fairly frequent topic of this blog, as an example of how integrative medicine embraces pseudoscience, does RCTs on it, and then either rejects the negative ones or spins them as being positive. TACT was basically a completely negative trial — except for one subgroup for whom the seeming positive result had a lot of red flags that suggested that it was probably spurious. Basically TACT showed that chelation therapy doesn’t work for coronary artery disease, with the possible exception of diabetics. Even if you accepted that result, taking it at face value, the honest thing to do would be to stop using chelation therapy in any patient with coronary artery disease other than diabetics. Of course, that’s not what happened. Believers still use chelation for everyone with heart disease.
In fairness, clinical trialists not studying woo can be prone to this very tendency. We’re all human, and when you’ve put a lot of work, time, and effort into a clinical trial, it can be very hard to accept a negative result and move on. Human nature demands that you dig as deep as you can to salvage something from all that work. Given that, my tendency is to counter Bothwell et al by suggesting that physicians’ faith in some medical treatments can be almost religious in nature, to the point where they lash out at negative clinical trials, no matter how well-designed.
The problem(s) with RCTs: More problems with doctors than RCTs?
Failure of adherents to pay attention to negative clinical trials is a problem, as Bothwell et al point out, which brings me to another complaint:
Even well-conducted RCTs sometimes failed to influence medical practice. In the late 1960s, the meticulously designed University Group Diabetes Program trial linked the antidiabetic drug tolbutamide with increased cardiovascular mortality. Yet tolbutamide prescriptions paradoxically increased as controversies over the trial’s conduct and interpretation persisted for more than a decade. A similar scenario occurred when the publicly funded ALLHAT (Antihypertensive and Lipid-Lowering Treatment to Prevent Heart Attack Trial) revealed in 2002 that generic thiazide diuretics were as effective as newer, expensive calcium-channel blockers and angiotensin-converting–enzyme inhibitors in treating hypertension. As these findings were contested by pharmaceutical manufacturers and skeptical physicians, sales of the newer antihypertensives grew faster than those of diuretics. Another 2002 RCT — a sham-surgery–controlled trial — defied conventional wisdom by showing no benefit of arthroscopic débridement for chronic osteoarthritis of the knee. Many orthopedic surgeons dismissed the results and continued performing the procedure, even as the findings were confirmed repeatedly.
I’m with Dr. Prasad again on this one, when he notes that failure to abandon treatments after large negative RCTs is more a failure to educate doctors how to evaluate evidence and a failure of third party payors to stop paying for such procedures. He’s particularly spot on here, noting that doctors’ refusal to abandon “debunked” treatments is:
…testimony to how hard it is to put the genie back inside the bottle once it is out, and if anything it should make us more worried about approving, using and paying for therapies that have not shown upfront benefit in RCT.
I’m also confused here by Bothwell et al. They just spent much of their article complaining about how RCTs have become the “gold standard” that trumps every other form of medical knowledge, a complaint that is not completely without validity but does tend to be overblown and, whether Bothwell et al realize it or not, used as a tool to argue for accepting less rigorous scientific evidence as the basis for adopting a treatment. Now they complain that physicians ignore the results of negative clinical trials, which is undeniably true more often than any of us would care to admit. I’ve even discussed such examples myself. My point here is not to deny that both can be true (RCTs are undeniably considered the “gold standard” of medical knowledge and doctors do from time to time reject RCT results they don’t like) but rather as a prelude to addressing another complaint by Bothwell et al, which basically boils down to a “Well, duh!” observation:
Yet RCTs have never monopolized medical knowledge production. A quick scan of the medical literature reveals that older methods, including case series and even case reports, continue to be valuable. New methods of observational research continue to emerge — for instance, using large databases of patients to produce comparative effectiveness data on various treatment outcomes relatively efficiently in settings of routine care. Physicians also continue to rely on physiological rationales in addition to empirical data. Coronary angioplasty and the stents that followed rose to prominence thanks not to a successful RCT but to the intuitive logic of the techniques and the compelling visual evidence provided by angiography.
Yes, less rigorous forms of clinical study are indeed valuable and frequently published in the medical literature, but they are not nearly as valuable in the way implied by Bothwell et all, namely, as Vinayak Prasad puts it, “MAKING A THERAPEUTIC RECOMMENDATION TO A PATIENT AND KNOWING YOU AREN’T SELLING THEM BULLSHIT.” (Capitalization his.) They can be valuable for hypothesis generation. They can be valuable to see if there is any evidence of an effect of a new treatment that would justify a full RCT. They can be valuable for looking for adverse reactions and complications. But for deciding definitively if a treatment works or not? Not so much. That’s not to say that they can’t be used for that. Sometimes we have no choice, for example, when for whatever reasons RCTs haven’t been done and/or can’t be done for ethical reasons to address a question. In those cases, which are not infrequent, less rigorous forms of evidence are all we have to fall back on until better evidence is available. It’s also hard not to mention here that if physicians can find reasons to justify rejecting the results of rigorous RCTs, relying on less rigorous studies will just give them a better excuse to ignore clinical evidence.
Also, doctors do continue to apply physiological rationales to justify treatments. Need I remind you that these physiological rationales are frequently incorrect? Doctors at the time thought they had a perfectly valid physiological rationale for using internal mammary artery ligation, for instance, to treat angina. When studied in an RCT, it didn’t work better than placebo. Dr. Prasad also notes another example, using stents to “open up” blockages causing stable angina (angina that comes on after a certain amount of exertion that isn’t changing). After all, the before-and-after pictures are so impressive! Look at how that blood vessel has been opened up! But:
If you want proof of this, see a recent Twitter string started by John Mandrola, where he questioned whether afib ablation requires a sham RCT to demonstrate QoL benefits beyond placebo. The response from EP doctors was fierce. Many tweets in reply showed images of afib that terminated after ablation, but of course, this is besides the point. No one doubts that you can terminate afib (a surrogate) in some patients, the question is whether this translates into an improvement in quality or quantity of life beyond the procedure itself. Same is true for stenting for stable coronary disease. No one doubt it opens the narrowed artery, but it doesn’t decrease MIs or mortality (COURAGE), and are gains in symptoms real, or simply an expensive placebo effect? (COURAGE did not have a sham control) A sham trial will tease it out. Sham controls aren’t needed for OBJECTIVE outcomes, but they are needed for SUBJECTIVE ones.
Which is what I’ve been saying about oncology trials. If the outcome you’re looking at is overall survival, that’s about as objective an outcome as you can imagine. Either you’re alive or you’re dead. Also, it’s unethical to use a sham control in many instances when the outcome being measured is so objective, which is why most oncology trials compare standard of care versus standard of care plus new intervention. Also, what Dr. Prasad doesn’t address is that some seemingly “objective” outcomes might not be entirely objective. For instance, relapse-free survival can be affected by how often and how carefully one screens for relapse, and interpretation of radiologic scans isn’t always 100% objective because human beings do it. That’s why blinding of radiologists reading scans for trial patients is important.
Unfortunately, Dr. Prasad also seems a little too blithe about the ethics of some RCTs for my taste:
Now what about sham trials of total knee arthroplasty for pain—in other words big surgeries. Well, I bet if we bucked up, wiped away surgeon tears, and ran them (control arm gets sedation and a long incision on the skin, and maybe some superficial hardware for them to palpate to simulate replacement), we would find that there may be a benefit for people with true joint instability, but maybe not for those merely with pain. Don’t get me wrong– I don’t know what such a trial will show, but it is entirely plausible that we will find that the procedure is no more than a sham operation for a large portion of customers. It will probably take another 10 years for orthopedic surgeons to muster the courage to do these trials, as it would threaten 30 billion a year in revenue. But such a trial would be a tremendous good for the public. Even if it confirmed benefit it would assure us that hundreds of billions and many complications were not in vain (and give us real $ per QALY figures not guesses). I however do not assume benefit here.
Which is reasonable, but greatly downplays the ethics of doing sham surgery-controlled RCTs of surgical interventions. Ethical issues in human subject experimentation are why in a lot of cases, we don’t have evidence from RCTs with a sham control group to support surgical interventions. I also can’t help but note that for objective outcomes sham controls aren’t necessary anyway. Dr. Prasad tells me so. Of course, most outcomes from such procedures are not, strictly speaking, objective: pain, stiffness, etc.
The problem(s) with clinical trials: They’re too slow!
Not all of the criticisms leveled at RCTs by Bothwell et al are invalid. Indeed, they make a good point that RCTs have become more bureaucratic and, as a result, very expensive, now requiring costly infrastructure for research design, patient care, record keeping, ethical review, and statistical analysis, noting that a single phase 3 RCT can cost $30 million. (Remember how I mentioned that I could do a decent-sized RCT for the $25 million that it cost to do the rat study that allegedly showed a link between cell phone radiation and cancer?) This is undeniably true. Unfortunately, it’s not always clear how to decrease the bureaucracy and still protect patient safety and enforce ethical standards, particularly given how the high cost of clinical trials provides a huge incentive to investigators and stakeholders to produce positive results. Although they don’t make a judgment on the 21st Century Cures Act (I describe it as nothing more than a false promise that loosening regulatory standards will deliver cures faster), they do correctly note that it is in part a reaction to these problems with clinical trials and would curtail the use of RCTs in drug and device development in the name of speed and increased efficiency.
One consequence of how slowly clinical trials are conducted is this:
One long-standing, possibly intractable, concern has been the discrepancy between the time frame of RCTs and the fast pace of innovation. In debating how best to evaluate CABG in 1976, surgeons complained that “just when we have accumulated enough data over a sufficient time period, we find that surgical technique has improved or medical therapy changes, or both, and conclusions no longer apply.” Major RCTs have often required many years for patient enrollment, follow-up, and analysis. In cases of rapidly evolving therapies, RCT results have seemed outdated before they were published. When the COURAGE (Clinical Outcomes Utilizing Revascularization and Aggressive Drug Evaluation) trial showed disappointing efficacy results for coronary angioplasty in 2007, the procedure’s advocates argued that the results were no longer relevant because the bare-metal stents tested in the trial had been replaced by newer drug-eluting stents. This logic, which assumes the superiority of any innovation, has created a setting in which trialists struggle to keep up with continuous innovations, similar to the “Red Queen” effect in evolutionary biology.
No doubt this is a problem, and we do tend to assume, often without justification, that new is better, particularly when it impacts our livelihood. So when a trial of old stents show they don’t work as well as believed, it’s not surprising that the counterargument would be that we have this nifty new and improved stent that really is better and must be working. Of course, by the time these new drug eluting stents are fully tested, there’ll be something else.
Then there are patient expectations and demand:
The AIDS crisis brought many tensions into stark relief in the late 1980s. Patients, frustrated that RCTs would delay approvals of antiretroviral drugs, demanded access before trials had been completed. Clinicians felt conflicted between their roles as physicians and as scientists. Activists won support for more flexible approaches to clinical research, including the use of surrogate end points, conditional FDA approvals, and parallel tracks to provide access to drugs outside of trials. Critics worried that the loosened standards undermined scientific rigor and encouraged a risky deregulatory agenda championed by the drug industry.
This issue lives on in the “right to try” movement, which has resulted in laws in many states that purport to provide access to experimental drugs to terminally ill patients off trial. Jann Bellamy and I have addressed how deceptive the claims behind these laws are and how they won’t really do anything to help terminally ill patients, but clearly part of the impetus for these laws, exploited brilliantly by the free market worshiping and libertarian-leaning Goldwater Institute, is the understandable frustration at how long clinical trials take, just as the activism of AIDS activists and victims in the 1980s and 1990s derived from the same fears.
Then, of course, there’s the development of “precision” medicine (formerly known as “personalized” medicine), which involves using various genomic markers and other biomarkers from individual patients to determine which treatment should be used. On the surface, this appears to be a major challenge to RCTs, but I’ve come to think that the challenge, while formidable, is perhaps not so insurmountable as critics of RCTs would like you to think. After all, it is quite possible to test standard-of-care against a precision medicine strategy designed to tailor the each patient’s treatment based on a set of biomarkers. This was done in the SHIVA Trial, to which I alluded in another post, and guess what? No advantage was observed in the “precision oncology” arm of the trial. That doesn’t mean precision medicine doesn’t have promise (the strategy used just might not have been optimal), but it does suggest that we should be a bit more circumspect about the promise of “precision medicine” and perhaps less pessimistic about the ability of scientists to develop new RCT methodology to study precision medicine approaches. RCTs are evolving.
Is the proposed solution worse than the disease?
There is no doubt that developments in medicine, particularly the genomics revolution, are placing strain on the standard method of doing RCTs and the infrastructure that supports it. Couple that with patient demand for faster drug approval (although I wonder what they’ll say if that faster drug approval process leads to more issues like Vioxx or even worse), and it’s clear that RCTs must evolve. I’m just not sure I can get behind what Bothwell et al say about how they should evolve:
The idea that RCTs would be the only authoritative arbiter to resolve medical disputes has given way to more pragmatic approaches. Experimentalists continue to seek new methods of knowledge production, from meta-analyses to controlled registry studies that can easily include large numbers of diverse patients. Observational methods are seen as complementary to RCTs, and new forms of surveillance can embed RCTs into the structure of data collection within electronic health records. RCTs are now just a part — though perhaps the most critical part — of a broad arsenal of investigative tools used to adjudicate efficacy and regulate the therapeutic marketplace. This status may continue to evolve with the recent turn (back) to personalized or precision medicine. As medicine focuses on the unique pathophysiology and coexisting conditions of individual patients, the applicability of the generalized data produced by RCTs will come under intensified scrutiny.
Before I provide my take, I can’t help but quote Dr. Prasad again about the issue of meta-analyses:
Uh, meta-analysis is most often pooled RCTs—so this makes no sense. You can’t say we don’t need potatoes because we have French fries—what are fries made of?
Yep.
But what about the rest? Controlled registry studies are useful for post-marketing surveillance but for determining what works and what doesn’t? Again, not so much. Observational studies, again, can be useful for tracking outcomes and adverse events, which can definitely complement RCTs, but for determining efficacy they aren’t that great. It’s hard not to interpret an underlying attitude in this article that the way to overcome the problems with RCTs is to start accepting less rigorous forms of evidence for efficacy, particularly in light of this introduction to one section:
RCTs have also unintentionally limited the producers of medical knowledge. When case reports constituted valid evidence of therapeutic efficacy, a single physician, drawing on clinical experience, could write an article that might change clinical practice.
They say that as though it’s a bad thing that doctors churning out case series don’t have nearly as much influence as they used to. It’s not. Most assuredly, it is not. Also, there are times when RCTs cannot be done, such as for rare diseases or for questions for which an RCT would be unethical, such as a randomized, controlled clinical trial of vaccinating versus not vaccinating. In those cases, we have to make do synthesizing lesser forms of evidence.
In the end, I can’t agree with Dr. Prasad that this article is some sort of subtle and dangerous threat to RCTs or that the NEJM is somehow intentionally trying to undermine RCTs as the primary method of clinical investigation. I do, however, agree with Steve that the answer is reform, not overthrow, and with Dr. Prasad that the solution to the problem of RCTs that no longer serve our medical needs as well as we require is to design better RCT methodology, not to start accepting less rigorous forms of evidence.

It’s time for true transparency of clinical trials data
Scott Gavura on November 8, 2012
Link
What makes a health professional science-based? We advocate for evaluations of treatments, and treatment decisions, based on the best research methods. We compile evidence based on fair trials that minimize the risks of bias. And, importantly, we consider this evidence in the context of the plausibility of the treatment. The fact is, it’s actually not that hard to get a positive result in a trial, especially when it’s sloppily done or biased. And there are many ways to design a trial to demonstrate positive results in some subgroup, as Kimball Atwood pointed out earlier this week. And even when a trial is well done, there remains the risk of error simply due to chance alone. So to sort out true treatment effects, from fake effects, two key steps are helpful in reviewing the evidence.
1. Take prior probability into account when assessing data. While a detailed explanation of Bayes Theorem could take several posts, consider prior probability this way: Any test has flaws and limitations. Tests give probabilities based on the test method itself, not on what is being tested. Consequently, in order to evaluate the probability of “x” given a test result, we must incorporate the pre-test probability of “x”. Bayesian analysis uses any existing data, plus the data collected in the test, to give a prediction that factors in prior probabilities. It’s part of the reason why most published research findings are false.
2. Use systematic reviews to evaluate all the evidence. The best way to answer a specific clinical question is to collect all the potentially relevant information in a structured way, consider its quality, analyze it according to predetermined criteria, and then draw conclusions. A systematic review reduces the risk of cherry picking and author bias, compared to non-systematic data-collection or general literature reviews of evidence. A well-conducted systematic review will give us an answer based on the totality of evidence available, and is the best possible answer for a given question.
These two steps are critically important, and so have been discussed repeatedly by the contributors to this blog. What is obvious, but perhaps not as well understood, is how our reviews can still be significantly flawed, despite best efforts. In order for our evaluation to accurately consider prior probability, and to be systematic, we need all the evidence. Unfortunately, that’s not always possible if clinical trials remains unpublished or are otherwise inaccessible. There is good evidence to show that negative studies are less likely to be published than positive studies. Sometimes called the “file drawer” effect, it’s not solely the fault of investigators, as journals seeking positive results may decline to publish negative studies. But unless these studies are found, systematic reviews are more likely to miss negative data, which means there’s the risk of bias in favor of an intervention. How bad is the problem? We really have no complete way to know, for any particular clinical question, just how much is missing or buried. This is a problem that has confounded researchers and authors of systematic reviews for decades.
Fortunately, it’s difficult to do a clinical trial without leaving any clues at all. You need to write a protocol, seek some sort of ethics board approval, find researchers (often physicians) to run your study, recruit patients, and finally, fund it. Assuming most investigators don’t set out to keep a clinical trial hidden from view, there may be other clues that find their way into the public domain. For example, at medical conferences there may be hundreds of academic “posters” and other presentations of research evidence – sometimes only the interim results of research studies. Only a fraction of this research ends up into the medical literature fully published. Sometimes data emerges in lawsuits, or in drug approval applications to regulators like the FDA, who may make this data available publicly. But searching for evidence shouldn’t rely on serendipity. To conduct the most accurate systematic review, we need a comprehensive system to track down every relevant clinical trial.
The simplest approach to finding relevant studies would be to register every single clinical trial conducted – ideally before the study begins, so that negative trials can’t subsequently be hidden without leaving any clues. Requiring a simplified registration at a central, international resource should not be too onerous a requirement for investigators planning to perform human experiments. Making this information publicly available would then allow researchers to determine how many trials had been performed, and could serve a starting point for a truly systematic review. It’s a simple and fairly elegant approach, so it should be no surprise that it’s been implemented already – albeit ineffectively. More that 25 years has passed since the first major call for a registry, yet even today we still don’t have an effective one. The pharmaceutical industry, led by GlaxoWellcome, started one voluntarily, but it was subsequently shuttered. (GlaxoSmithKline has just recently announced plans to return to this level of transparency, to their credit.) The FDA passed legislation to require trial registration in 1997, and mandated the registration of “federally or privately funded clinical trials conducted under investigational new drug applications“. The well-known clinicaltrials.gov launched in 2000. In 2007, requirements were expanded to include all clinical trials (with the exception of early “phase 1” studies) for all products subject to the FDA’s regulation. There have been external pushes to require registration, including the International Committee of Medical Journal Editors who implemented this requirement in 2005 for any study considered for publishing. This looked like a good strategy – researchers that avoided registering would find themselves cut out from subsequently publishing in the world’s major medical journals. There was also the World Health Organization’s mandate to register in 2006, and then the World Medical Association’s Declaration of Helsinki, which establishes ethical principles for clinical trials, which was amended in 2008 to require that “Every clinical trial must be registered in a publicly accessible database before recruitment of the first subject.”
In light of this consensus throughout the research and clinical community that registries are a Good Thing, how effectively are they working? Not as well as you might think – only one-fifth of eligible trials have actually been registered, based on an 2011 analysis. Don’t pin this all on the pharmaceutical industry. Among the different groups of investigators, industry-sponsored research was more likely to be registered than government- or other-funded research. It seems no group is particularly good at walking the talk of registries, despite their obvious benefits to medical science. In response, a new bill has proposed further penalties, including requiring repayment of any federal (NIH) funding, and forbidding further funding, for any group that does not publish its data fully. Another proposed bill will impose FDA requirements on all international trials used to support any application made to market a drug in the USA.
Still, there continues to be a lack of motivation on the part of some regulators to embed the need for registries into their processess. Just recently in Canada, a “web-based list” of regulator-approved trials was announced by Health Canada, which was heralded as out-of-touch and underwhelming – no mandatory registration, no enforcement, and no intent to make the results of public trials available. Compared to the FDA, it’s circa-1990’s transparency and policy thinking.
Access to research information, ideally patient-level data, is what’s necessary fairly evaluate risks and benefits of therapies. This allows other researchers to look at the evidence themselves, and potentially validate findings. It’s the ultimate in data transparency. To the credit of one member of Big Pharma, GlaxoSmithKline (GSK) has now promised this level of transparency. And last week, in lauding GSK’s announcement, Fiona Godlee, editor of the British Medical Journal announced the BMJ will require this data disclosure – from all researchers as of January, 2013, for any trial published in their journal:
And amid the plaudits, a moment of doubt. Surely what this apparently brave and benevolent action really serves to highlight is the rank absurdity of the current situation. Why aren’t all clinical trial data routinely available for independent scrutiny once a regulatory decision has been made? How have commercial companies been allowed to evaluate their own products and then to keep large and unknown amounts of the data secret even from the regulators? Why should it be up to the companies to decide who looks at the data and for what purpose? Why should it take legal action (as in the case of GlaxoSmithKline’s paroxetine and rosiglitazone), strong arm tactics by national licensing bodies (Pfizer’s reboxetine), and the exceptional tenacity of individual researchers and investigative journalists (Roche’s oseltamivir) to try to piece together the evidence on individual drugs?
Data secrecy has been the norm for decades. But perhaps it’s worth asking: Just who benefits from this data paternalism? After all, who needs that information more than those prescribing those treatments – and the patients already taking the drugs in question? GSK’s announcement was an important one for a pharmaceutical manufacturer, and they have just announced their support for the BMJ’s transparency requirements. Whether this will improve the trust that health professionals have in pharmaceutical-industry sponsored research remains to be seen. To be clear, the entire industry isn’t on board yet. In time, a genuine demonstration of transparency could help the pharmaceutical industry recover from the damage it repeatedly self-inflicts on its own credibility.
I’m cautiously optimistic, with GSK’s announcement and the BMJ’s action, that there may be some momentum building. In September, the European Medicines Agency, the EU equivalent to the FDA, announced plans to proactively publish clinical trial data, and remarkably, allow access to full data sets. Ben Goldacre’s latest book, Bad Pharma, which provides a comprehensive yet accessible summary of the evidence and the consequences of a lack of transparency, has drawn widespread media attention in the UK and seems to be getting some traction in his calls for a fix. It will be interesting to see the North American reaction to his book, when it appears here in 2013.
Conclusion
We know evidence is missing or inaccessible. We know how to fix it. A lack of data transparency compromises the evidence base, prevents the best science-based care, and ultimately, does a disservice to patients. The benefits of open data on how we deliver medicine are obvious. Yet the foot dragging and hand waving continues, from manufacturers, regulators, and other interest groups. It’s shameful, and it needs to change.

Marty Makary and Vinay Prasad embrace a “more fluid concept of evidence” at the FDA
Last Thursday during a roundtable on stem cell therapies, new FDA Commissioner Marty Makary referred to EBM levels of evidence as an artificial and dogmatic construct. Apparently Dr. David Katz’s “more fluid concept of evidence” now reigns at the FDA—selectively.
David Gorski on June 9, 2025
Link
I’m going to start this post with two quotes for you to contemplate. First, during the FDA Roundtable on Cell and Gene Therapy held last Thursday, I witnessed the new FDA Commissioner Marty Makary saying near the end of the proceedings:
I think we can learn a lot from individual experiences. We tend to talk in the lexicon of clinical medicine about level 1, 2, and 3 evidence. It has become this sort of dogma that we can only do things with evidence, and the evidence has been categorized in this artificial framework of level 1. 2. and 3, one defined as randomized control trials. Level one is better than two is better than three.
Now, compare and contrast the above to what Dr. David Katz said at a conference held at Yale University in April 2008 entitled the 1st Annual Integrative Medicine Scientific Symposium. By way of background for those who haven’t been reading SBM long enough to have read what Steve Novella, Kimball Atwood, and I wrote about this symposium as soon as video of the proceedings had hit YouTube, Dr. Katz is a fan of “integrative” medicine, which we at SBM like to characterize as “integrating” quackery like acupuncture and homeopathy with science- and evidence-based medicine.
The Five Horsemen of the Quackocalypse.
Here’s the money quote:
I think we have to move beyond the results of RCTs in order to address patient needs, and to do that I’ve arrived at the use of a more fluid concept of evidence than many of us have imbibed from our medical educations…[Referring to an anecdote of a patient with chronic pain Dr. Katz went on.]…Now, we don’t want you on narcotics any more than you want to be on narcotics. We initiated a course of acupuncture and over the next two to three months weaned him off narcotics. He was pain-free on acupuncture and subsequently transitioned into homeopathy. Now, I don’t care to get into a discussion of how or even whether homeopathy even works, but this guy had tried everything.
This particular talk caused quite the kerfuffle among those of us who had long been trying to promote better science in medicine through the concept of science-based medicine. Three of us at SBM piled on, but so did David Colquhoun and Dr. R. W. Donnell, among others. Indeed, ever since, a “more fluid concept of evidence” has been one of our go-to sarcastic terms for the sorts of “evidence” that alternative medicine quacks routinely traffic in.
More saliently, note the similarity between what Dr. Makary said four days ago and what Dr. Katz said 17 years ago, specifically that anecdotes can trump randomized controlled clinical trials (RCTs). It goes beyond that, though. Both portray evidence-based medicine as “dogma,” with Dr. Makary calling it that explicitly. Although the quote above by Dr. Katz doesn’t express such a view, other parts of his talk back then most definitely did, such as when he referred to the evidence used as the basis of evidence-based practice as “indoctrination.” It’s a message that’s common to quacks, antivaxxers, and all manner of science deniers, portraying science that they don’t like as a religion.
Dr. Vinay “RCTs or STFU” Prasad joins the party
Sitting just two seats away from Dr. Makary was Dr. Vinay Prasad. We’ve criticized Dr. Prasad many times on his blog for his methodolatry, defined as the obscene worship of the randomized controlled clinical trial as the be-all and end-all of clinical investigation and the only valid method to use, all in order to weaponize the evidence-based medicine paradigm against COVID-19 mitigations, including “lockdowns,” masking, and vaccines, particularly boosters. Indeed, he routinely used to refer to his philosophy as “RCTs or STFU” (randomized controlled trials or…well, you know) to the point that before he was appointed to head the Center for Biologics Evaluation and Research (CBER), the center in the FDA responsible for evaluating and approving biologics such as vaccines, stem cell therapies, and gene therapy, he had even started to fall for the deceptive antivax narrative that tried to portray all the vaccines in the childhood schedule as dangerous because they hadn’t all undergone RCTs versus saline placebo, a deceptive narrative that new Secretary of Health and Human Services Robert F. Kennedy Jr. had been pushing for years (along with antivax lawyer Aaron Siri) and that has infested HHS since he was put in charge.
Yet there was Dr. Prasad, nodding along with his boss Dr. Makary as his boss intoned that RCTs aren’t always necessary and questioned the construct of “levels of evidence” as “artificial” and dogmatic, asserting that anecdotal evidence can be “causal until proven otherwise.” Dr. Makary, like Dr. Katz 18 years ago, even cited anecdotes, specifically that of a glioblastoma patient who had survived much longer than the average glioblastoma patient and another from parents who believed that certain that food dyes had caused their child’s aggressive behavior.
Indeed, Dr. Prasad appears to be all-in. Early in the roundtable, Dr. Prasad boasted:
How will we make our judgments? We will rely on, as Dr. Makary says, gold standard science and common sense. What does that mean practically? I’m often asked: What do you think about surrogate endpoints that do not intrinsically matter to patients but which often correlate with those that intrinsically matter? What do you think about overall survival? How do you weigh these two things? The answer is: We’re interested in both. We will rapidly make available therapies at the first sign or promise of biomedical success or action, but we’re also going to follow up overall survival and quality of life on the back end to ensure that we’re accomplishing what we think we’re accomplishing.
I’m often asked: Are you interested solely in randomized controlled clinical trials, or are you open to other methodologies? The answer is, we’re interested in all of the above: randomized controlled trials, whether or not people can serve as their own individual controls. We’re interested in target trial emulation, some of the more sophisticated epidemiological techniques, and we developed something called the iceberg plot, which develops patients against prior therapies, and something we call a parachute trial, which is a way in which we protect patients from old and dilapidated control arms, where we have efficacy.
This is “RCTs or STFU” How fluid Dr. Prasad’s view of medical evidence has become!
In fairness, I’ve long been saying that RCTs cannot be the be-all and end-all of medical evidence, given that medical ethics precludes trials that would be unethical, forcing us to rely on a preponderance of “lesser” evidence and that sometimes RCTs are simply impractical or so expensive as to be prohibitive. I’ve long viewed Dr. Prasad’s “RCTs or STFU” as methodolatry and EBM fundamentalism. That being said, it is amazing to see how fast Dr. Prasad has pivoted from “RCTs or STFU”—or, as I’ve sometimes characterized it, “RCTs über alles“—to Katzian fluidity with respect to medical evidence.
In addition, apparently Dr. Prasad’s “parachute trials” don’t apply to protecting children from having to be randomized to saline placebo and vulnerability to a vaccine-preventable disease when a second generation product is being developed of a vaccine with known efficacy. It’s also more hilarious that he claimed credit for developing a “parachute trial.” I did a PubMed search and could find only three references authored or coauthored by Dr. Prasad about “parachute trials.” In each of the references, Dr. Prasad is actually very critical of other physicians who liken RCTs of certain interventions versus placebo to the “parachute trial” proposed in a classic 2003 satirical paper that proposed an RCT of parachutes. The intent of the satire was to use a ridiculous example (that of parachutes preventing death in people jumping out of airplanes) to demonstrate why some clinical trials versus placebo are inherently unethical. Indeed, Dr. Prasad’s papers sport titles like, Most medical practices are not parachutes: a citation analysis of practices felt by biomedical authors to be analogous to parachutes, The use and meaning of the parachute metaphor in biomedicine: a citation analysis of a systematic review and a randomized trial of the parachute for freefall, and Where are randomized trials necessary: Are smoking and parachutes good counterexamples? The messages from all of these publications are that Dr. Prasad thinks that the parachute metaphor is overused and overblown and that RCTs of medical interventions likened to “parachutes” are not unethical because these interventions are not as effective as believed.
For instance, here’s Dr. Prasad in 2018:
Although there is widespread interest regarding the BMJ paper arguing that randomized trials are not necessary for practices of clear benefit, there are few analogies in medicine. Most parachute analogies in medicine are inappropriate, incorrect or misused.
Here’s Dr. Prasad in 2022:
Why, then, do people often assert that RCTs are unfeasible, despite the danger associated with adopting therapies without randomized data? How is it appropriate to use smoking and parachutes as counterexamples? Medicine cannot be likened to a parachute; our patients do not leap from planes, our treatments are not as successful as hitting-the-silk while falling from the sky, and demonstrating effectiveness in a patient group is much more complex than pulling a rip cord. As a result, almost everything in biomedicine can be randomized; in most cases, there is equipoise. A persistent bias in biomedicine is that more expensive, intrusive and novel treatments must improve outcomes; nevertheless, the only way to remove this bias is to confront it using empiricism. Ignoring evidence and relying on heuristics and personal judgement in the face of empiricism may result in a loss of credibility, a stalling of innovation and a loss of public confidence in our medical initiatives.
See what I mean? It’s fair enough that a lot of interventions portrayed as “parachute-like” don’t have evidence quite so strong as to justify the label. It’s even fair enough to point out that sometimes advocates of various treatments are too quick to invoke the parachute” metaphor. That’s not what Dr. Prasad has been about, though. He’s been about “RCTs or STFU,” and the parachute metaphor gets in the way of that argument. All of this is why I also find it hilarious that Dr. Prasad claims in the quote above to have developed the concept of a “parachute trial” to spare human subjects outmoded control treatments; unless I’m missing a publication or podcast statement, he’s done quite the opposite; yet a couple of the speakers invoked “parachutes” as though their stem cell therapies were the equivalent of parachutes for which an RCT would be unethical.
Moreover, a parachute RCT has been done. (Seriously, read the trial.)
It’s equally hilarious that Dr. Prasad seems to think he’s the first to have thought deeply about many of the concepts in clinical evidence that he trots out (other than the iceberg plot, which he does appear to have originated). Moreover, much of what Dr. Prasad said is nothing new. For instance, one phrase stood out: Make available therapies at the first sign or promise of biomedical success or action. These are basically surrogate endpoints (i.e., biochemical changes associated with activity of a drug or other endpoints associated with overall outcome, such as overall survival in an oncology trial). Such endpoints have been the very basis of accelerated approval programs, such as for HIV drugs or in oncology, for decades. One can’t help but note that Dr. Prasad has been very critical of even before he became a COVID-19 contrarian. Yet here he is, championing what sounds very much like an accelerated approval program on steroids for stem cell therapies, while others invoke the federal “right-to-try” law.
Remember, as I have written extensively, “right-to-try” is a cruel sham and an illusion. It sells false hope and paved the way for terminally ill patients to access experimental drugs after only phase I trials, a concept that easily can lead to the exploitation of desperate patients on par with that demonstrated by cancer quack clinics in Bavaria. Indeed, President Trump has claimed that right-to-try, passed during his first administration, has saved thousands of lives. It hasn’t. It’s doubtful that it’s saved even one life.
It would appear that Marty Makary and Vinay Prasad have embraced David Katz’s “more fluid concept of evidence”! At least, they have when it comes to their boss RFK Jr.’s favored treatments, like the topic of the roundtable, stem cell therapies. Indeed, as you will see, this “more fluid concept of evidence” is very selectively applied. Let’s take a look at the rest of the roundtable. Be warned. It’s nearly three hours long. I will confess that I sped through parts of It or just read the YouTube-generated transcript when I got bored, but the overall message was very clear: RFK Jr. wants stem cell therapies approved and that he will get stem cell therapies approved, even if Makary and Prasad must embrace a more fluid concept of evidence and like it.
For those of you who can stomach it, here’s the whole roundtable, all three hours of it, particularly the part where Dr. Makary refers to Dr. Prasad as a “medical and scientific genius.”
First, a bit of background.
The sordid history of stem cell quackery
It is impossible to view this roundtable without briefly considering the history of stem cell clinics and the long history of FDA laxness when it comes to the regulation of stem cell therapies. It’s something that we at SBM have written about and lamented for a long time. Quite contrary to the portrayal in this roundtable of the FDA has an overweening, almost fascistic, bureaucratic presence that hampers innovation and getting the cures to the people, the FDA has been slow to shut down even stem cell clinics that are selling obvious quackery. Indeed, dubious stem cell clinics have sold stem cells to treat all manner of diseases for which the owners of the clinics have presented no RCT evidence—often no evidence at all except for anecdotal evidence—for the efficacy and safety of their concoctions.
Such clinics have, however, registered clinical trials on ClinicalTrials.gov, usually with no control group and questionable science behind them, as marketing tools. They’ve found willing dupes in academia to team up with them to set up “pay-to-play” clinical trials of dubious stem cell therapies for autism, a common condition targeted by stem cell quacks, many sold with overblown or outright fraudulent promises of great benefit, including that most suspect claim of all, anti-aging. Meanwhile, as has been documented by actual stem cell scientist Paul Knoepfler and myself, the sales techniques utilized by these clinics have more in common with a cross between techniques used by used car salesmen doing the “hard sell” and traveling snake oil salesmen touting “miracle cures” in the 19th century. Remember, to advocates of stem cell therapies, stem cells are basically magic. They can cure anything. While it is true that stem cell therapies do hold a lot of promise given the ability of these cells to differentiate into different tissues, the hype often far surpasses the reality.
Indeed, I first took an interest in dubious stem cell therapies over a decade ago, when hometown hockey hero Gordie Howe was enticed into traveling to Mexico for a dubious stem cell treatment sold by a company called Stemedica and touted as able to repair the neurological damage that he had suffered as the result of a severe stroke. There were a number of huge red flags in this story (documented in the links earlier in this paragraph), including the company administering the treatment in Mexico under the auspices of a highly suspect “clinical trial” because Howe was found not to be eligible for clinical trials in the US and the company paying the tens of thousands of dollars for the treatment because Gordie Howe was famous and it was good marketing. And, boy, was it ever great marketing! The media went wild, and pundits such as Keith Olbermann slurped up Stemedica press releases and carefully curated videos of Howe as if they were delivered from on high, producing interviews that read like infomercials for Stemedica. (Indeed, Olbermann actually used his platform to attack yours truly over my posts about the story.)
Now let’s look at who spoke (besides Drs. Makary and Prasad) and what the message was.
Is the FDA too harsh on stem cells and gene therapy?
Looking over the list of speakers featured at the roundtable, besides the five horsemen of the Quackocalypse—CMS Administrator Dr. Mehmet Oz, HHS secretary RFK Jr., FDA Director Dr. Marty Makary, NIH Director Dr. Jay Bhattcharya, and CBER Director Dr. Vinay Prasad—I had to admit that I was unfamiliar with most of these people. They are legitimate scientists, but they are also, by and large, very bullish on stem cells and rather dismissive of concerns about safety. A common theme is that regulation will stifle innovation, which is true to an extent, but lack of regulation also leaves patients the victim of charlatans, which is a largely the situation with respect to stem cell clinics now. Again, this whole roundtable was three hours in length; so I can only hit some of the “highlights” (or lowlights) that caught my attention.
For example, here’s Dr. Carl June, Director Center for Cellular Immunotherapies at the Perelman School of Medicine, University of Pennsylvania, who has an impressive record pioneering the development of CAR-T cell therapy. In brief, he raised the specter of “offshoring” the translation of basic science into actual stem cell and gene therapies. While this is a concern, he mentioned China, which has notoriously lax standards with respect to human experimentation and the regulation of, for instance, traditional Chinese medicine herbs and various pharmaceuticals. Do we want to race to the bottom with China?
Let’s see what Dr. June proposes, as it mirrors what a lot of the other speakers advocated for:
If we do not modernize our regulatory approach, we risk losing our leadership and undermining the long term viability of our biopharma industry. History shows that overcautious regulation can stifle progress. Bone marrow transplantation, where many of us started, was pioneered in the 1980s with only local IRB approvals outside of FDA oversight, and the field improved from initial mortality rates of over 20% to under 1% while improving efficacy. Had those high risk, early stage trial been subject to today’s FDA rules, they would have likely been halted, underscoring tha trigid early oversight can hold back promising advances. In my view, the FDA need not regulate early stage trials of autologous gene-modified cell therapies. Note here that I’m not referring to allogeneic products give to patients. I’m speaking of autologous products like hematopoietic cells that can be run under local IRB supervision. I propose a two-tier model like China does, where early models proceed under IRB approval and health review in China is required only once a therapie shows promise and is ready for larger multisite trials. In China, investigator-initiated studies begin without upfront authorization from the national regulatory bodies. Only if a product shows promise do they move into the standard regulatory process, which mirrors the FDA process.
This all sounds fairly reasonable as far as it goes. I note, however, that Dr. June comes out of academia and a standard scientific model of development of drugs, biologics, and other medical therapies. He assumes good faith. He also seems not to realize how easy it is for dubious clinics to set up compliant, rubber-stamp IRBs who will approve almost whatever they want to do, as per the examples of Drs. Stanislaw Burzynski and Mark Geier, who set up institutional review boards (IRBs) packed with their friends and cronies to approve their scientifically dubious and unethical studies, thus giving them the appearance of legitimacy. A two-tier system like the one proposed by Dr. June would seem desirable to me only if HHS also increased the stringency of its regulations covering IRBs or, at the very least, bothered to enforce the regulations already on the books.
Of course, one can’t ignore the context here, either. The current administration’s FY2026 budget proposal included massive cuts to the budgets of the FDA, CDC, and NIH. How is the FDA going to implement such modernization, even if such modernization is reasonable and desirable, given that if the administration’s budget passes Congress with those massive cuts intact it will have little or no funds to do so?
A recurring theme in all these talks was how much speed was necessary and the invocation of rare diseases as justification for “right-sizing” the current regulatory apparatus, which, depending on the speaker, involved proposals like those of Dr. June, such as Dr. Donald Kohn, Distinguished Professor of Microbiology, Immunology and Molecular Genetics, Molecular and Medical Pharmacology, and Pediatric Hematology/Oncology at UCLA, who complained about the regulatory hurdles for gene therapy and the difficulty commercializing new gene therapies and proposed steps to make the production of plasmids used in gene therapy less expensive. Others emphasized similar themes, some with compelling personal stories of family members with diseases.
It was, however, rather interesting to me, as it was to the aforementioned Dr. Paul Knoepfler, that most of these speakers didn’t even mention stem cell therapies all that much. (It was also puzzling to me how Dr. Knoepfler seemed to view this roundtable far more positively than it deserved, although in fairness he now appears to be having second thoughts.) Rather, they were talking about gene therapies, in particular gene editing technologies like CRISPR. In that context, Dr. June’s proposal of IRB-only approval for autologous therapies that are gene-altered, which presumably includes cells in which a gene has been modified by CRISPR, rather concerning to me given the current regulatory laxity with respect to IRBs in this country.
One speaker was a transplant surgeon named Dr. Jayme Locke, Vice President, Medical Development Xenotransplantation, United Therapeutics and an adjunct professor of surgery with a clinical practice at NYU Langone Health. Her work is in xenotransplantation, which is transplantation of an animal organ, often from an animal that has been genetically manipulated to make tissues organs more immunologically compatible with humans. It is an approach that could, if the challenges in overcoming the immune response to an organ from another species could be overcome, has the promise to open up the supply of organs in patients with organ failure on transplant lists:
What I would encourage the FDA and others is to not hold xenotransplantation and these other therapeutics to a standard that is different than what we hold allotransplantation to. Remember the patient voice. They want that right to try. Every day in human-to-human transplantation, I have patients who sign up to take organs that have hepatitis C, that have all these other risks, some known and some unknown, because they want the right to try, the right to live. I would just encourage us to not hold these therapies to a different standard. I would say that the thing that we must know and keep in mind is the patient’s voice. I thin it is critical to everything that we are doing, and I think we have to listen with the intent to understand, and I think that in many ways what our patients want is what the Trump administration did during their first administration, which is they want the right to ry. They want to feel that they have hope, that they’ve been afforded an opportunity.
This all sounds very compelling on the surface, but it is also our job as physicians, scientists, and clinical investigators to protect patients from experimental treatments in which the risk likely outweighs the potential benefit. Dr. Locke discussed a patient for whom she did xenotransplantation of a kidney from a genetically modified pig. The transplanted kidney lasted only four months before being rejected, but the patient viewed the effort as a success, even though the kidney had to be removed. To be honest, when Dr. Locke asks that xenotransplantation not be held to a “different standard” than allotransplantation (human-to-human transplantation), I can’t shake the feeling that that is exactly what she is asking, only with lower standards for xenotransplantation.
Again, I strongly suspect that all of these accomplished scientists and advocates calling for less FDA regulation of stem cell therapies were failing to see the elephant in the room, namely the stem cell quackery being sold at high cost by thousands of grifting clinics all over the country—all save one speaker, that is, the only one who provided a reality check on all the stem cell boosterism.
The real purpose of this roundtable: Lower the bar to approve stem cell quackery
Many of the issues brought up by the other speakers were not unreasonable. They are also ideas for altering FDA regulatory standards in response to new science that have been floated around and discussed for years, if not decades. It can indeed be argued that the FDA and its regulations do need modernization, given the era of genomic medicine and cell therapy and the incompatibilities between some of the treatments derived from these new sciences with the traditional pharma drug development model. The problem, again, is that the discussion at this roundtable almost completely ignored the context, including who RFK Jr. is and what he’s advocated over the years. Make no mistake, RFK Jr. is not about rationally “modernizing” the FDA per se. Rather, he’s about selectively applying different standards of evidence to facilitate a specific outcome. In brief, he wants to make it easier for charlatans to market quackery that he favors; e.g., bogus stem cell therapies, chelation therapy for autism, and unproven supplements. In parallel, he wants to make it more difficult for science-based treatments and preventatives (especially vaccines, to be approved, through the weaponization of the evidence-based medicine paradigm requiring RCTs even when they might be unethical, impractical, or unnecessary. In other words, he wants selectively alter standards of evidence at the FDA to favor his preferred “make America healthy again” (MAHA) remedies.
Only one of the speakers dared address this elephant in the room, stem cell quackery, although he did it without mentioning RFK Jr. or MAHA, which was probably wise from a strictly political standpoint. That one speaker was Sean J. Morrison, PhD, an investigator at the Howard Hughes Medical Institute and the director of the Children’s Research Institute at UT Southwestern in Dallas. He also serves as Chair of Public Policy Committee, International Society for Stem Cell Research. I’m going to quote his entire statement, because his is the only cautionary voice:
I just want to—I want to focus on a part of this issue that hasn’t been addressed so far. You’ve had really outstanding ideas this morning from serious people with deep insights and clinical evidence of safety and efficacy. But there area slso bad actors in this space who are attempting to sell essentially snake oil, to rip off desperate patients by capitalizing on the promise that you hear from the research in this room, but who are instead selling snake oil. These companies never do clinical trials. They have no plans to do controlled clinical trials. They are operating illegally, but it’s estimated that there’s 2,000 companies like this in the United States. They commonly claim to be able to cure diverse medical conditions with one-size-fits-all therapies, even conditions for which there is no plausible scientific rationale for stem cell therapy. For example, there’s no scientific rationale for how umbilical cord blood cells or placental cells could be used for the treatment of Parkinson’s disease or Alzheimer’s disease or orthopedic conditions or neuropathies. Yet there’s hundreds of companies out there that are selling to Americans making that claim.
Americans really need the FDA to protect them from those people. “Buyer beware” is not an effective strategy when it comes to experimental therapies, because even most physicians don’t have the expertise to evaluate which experimental therapies really have merit and which ones don’t. The companies, what we’ve learned is that the companies that ignore FDA regulations also commonly ignore good manufacturing processes. Scores of people have become septic as a result of injection of thes products. So I’ll just end by saying that the challenge here is to find ways of accelerating the development of real therapies with sound scientific rationales, clinical evidence of safety and efficacy, without deregulating to the point where you open the doors for the bad actors to sell snake oil to Americans. It would undermine confidence in the agency, and it would undermine confidence in the field in general.
Preach it, Dr. Morrison!
Unfortunately, Dr. Morrison’s wise warning was almost entirely ignored, and basically no discussion occurred subsequently regarding the entirely valid points that he had made. Instead we were treated after the last speaker to Dr. Makary referring to the EBM levels of evidence as “artificial” and “dogmatic.” I would counter by saying that the scientific method is “artificial.” Yet it works. After that, Dr. Bhattacharya touted the NIH-funded work that led to the recent gene therapy for sickle cell anemia, none of which occurred under his watch, and, given the proposed budget cuts to the NIH, similar advances are much less likely under his watch. Then we got Dr. Oz talking bout how stem cell therapy is MAHA because it gets at the “root cause” of disease, which is somewhat, but not entirely true, and real stem cell science has nothing to do with the other quackery and antivax nonsense promoted by MAHA.
When it came to RFK Jr.’s turn to give his closing remarks, I must admit to having been fairly nauseated as he praised the panel, particularly Drs. Makary, Bhattacharya, and Prasad as brave “renegades” who were “censored” during the pandemic, and how he likes to go to dinner with them and even on vacations because they all “enjoy each other’s company.” He then, even more nauseatingly, invoked his uncle President John F. Kennedy and his Presidency, going on to claim that only 3% of people had a chronic disease 60 years ago, while today 60% do. This is, of course, a narrative that leaves out a lot of relevant information. For instance, JFK’s presidency occurred near the tail end of the postwar baby boom, when only 12% of the population was 65 or older in 1963, compared with nearly 18% now. Diseases due to smoking have declined because the proportion of the population who smoke has declined dramatically since then.
“If R.F.K. Jr. makes the statement that more people are dying of chronic diseases now than in Jack Kennedy’s era, that’s undoubtedly true — we’ve got twice as many people, and a much larger chunk are old folks who have much higher chronic disease rates,” said Kenneth Warner, dean emeritus of the University of Michigan School of Public Health.
“Does that mean we’re doing worse than back then?” he added. “Absolutely not.”
And:
“When you look at chronic disease, you’re going to see that the diseases largely driven by cigarette smoking have declined since J.F.K. was president,” said Dr. JoAnn Manson, a professor of medicine at Harvard Medical School.
“But the chronic conditions driven by obesity or diet or sedentary lifestyle have increased,” she said, “including cardiometabolic diseases like Type 2 diabetes.”
That’s partly why progress toward improved health has stalled, she said.
Then, RFK Jr. said this:
Mehmet points out when my uncle was president we spent zero on chronic disease in this country. Now we spend $1.7 trillion.
One notes that Medicare and Medicaid only came into existence in 1965, nearly two years after JFK’s assassination. So, yes, during JFK’s presidency, the federal government likely paid very little to take care of chronic disease, because back then healthcare funding was nearly all private, paid for out-of-pocket or by employer-provided health insurance plans, with relatively little paid for by the federal government by comparison. The burden of chronic disease was thus borne by health insurance companies and individuals.
RFK Jr. also gives away the game later in his remarks:
President Trump has given us an executive order for every regulation put in place we have to get rid of ten. So I would solicit right now a list from all you of any regulations you think we ought to be getting rid of because we need to get rid of a lot of them in order to do the things you want to do.
There you have it. The real purpose of this roundtable is a precursor, a step on the way to radical deregulation of the stem cell therapies that RFK Jr. likes.
In fact, RFK Jr. had given the game away a week and a half before the FDA roundtable during an appearance on an episode of The Ultimate Human with Gary Brecka podcast. During the interview, RFK Jr. actually acknowledged that there are stem cell “charlatans” out there. (Really? I had never noticed.) In the same podcast, RFK Jr. (very) briefly related how he had gone to Antigua in order to receive stem cells for his spasmodic dystonia, the neurologic voice disorder that makes his voice the way it is, asking:
And that if you want to take an experimental drug, that you can do that. You ought to be able to do that. You shouldn’t have to go to Antigua to get stem cells, which I had to do for my throat.
Right. And they helped me enormously. Why did I have to go to Antigua for that?
This revelation, the first time that I’m aware of that RFK Jr. has admitted to having received dubious stem cell therapy, explains a lot. I also can’t help but note that, if anything, RFK Jr.’s voice has been sounding worse to me now compared to a few years ago, which is why I question his assessment that it had helped him. Here’s where he gives the game away, though, and reveals the real purpose of the roundtable, appropriately enough, right after an add for a supplement, liposomal NAD (no, seriously, you can’t make stuff like this up):
And you know, I thank you for what you’re doing and we’re going to end the war at FDA against alternative medicine. Thank you. The war on stem cells, the war on chelating drugs, the war on peptides, the war on anything that is not going to make big pharma money.
Just as he said he would, right after he had bent the knee last summer to Donald Trump and kissed his ring in return for a high-ranking health position in the Trump administration. Stem cells quackery is just part of his agenda for deregulation of quackery. But what about the charlatans? Well, what about them?
Per RFK Jr:
Our position is that FDA has a job just to do the science on these kind of issues and then tell the public what they’ve learned from the science, but not tell people and not tell physicians what they can and cannot prescribe.
And:
You know, we don’t want to have the Wild West. We want to make sure that information is out there, but we also want to respect the intelligence of the American people, the capacity of people who explore the outcomes that are going to benefit them the most. And of course, you’re going to get a lot of charlatans, and you’re going to get people who have bad results, but ultimately, you can’t prevent that either way.
And:
And that people may misuse them or stem cells or hyperbaric chambers for people for, you know, it’s not the government’s purview to tell people they cannot have access to those things because what they’re giving us access in the narrow range of drugs that they want to, products that they want to restrict us to are not making us healthier. We ought to rely on democracy and the good sense of the American people and the drive that we all have to take care of ourselves and God’s gift to us of a healthy body that has its own set of defenses that we need to respect.
Funny how RFK Jr. didn’t say any of this in his closing remarks at the FDA roundtable, in contrast to this podcast, where he lets his quack flag fly high. To paraphrase RFK Jr.: Sure there’ll be quacks and charlatans, but so what? Who cares? It’s not the FDA’s job to shut them down. It’s not the FDA’s job to tell doctors what they can and can’t do. It’s just up to the FDA to put the “science” out there and then let the buyer beware! Health freedom über alles!
You also need to understand some very important additional context. What RFK Jr. means by “gold standard science” and what scientists mean by good science tend to be related only by coincidence. As I like to say, RFK Jr. wouldn’t know good science if it bit him on the posterior. As an example, just look at the MAHA Report published a couple of weeks ago, which cherry-picked and misrepresented the science, apparently even using AI to write significant parts of the report, which included hallucinations of references that didn’t exist and studies that were never done.
This brings me back to Drs. Makary, Prasad, and Bhattacharya, especially Dr. Prasad. (I leave out Dr. Oz because he was always a quack, dating back to the 1990s, and so could be counted on to eagerly jump aboard RFK Jr.’s grift train if given the chance.) From my perspective, these pandemic-era self-proclaimed avatars of rigorous science who were “censored,” “canceled,” and “persecuted” by the medical establishment for advocating “RCTs or STFU” and related exaggerated and deceptive attacks on the science behind COVID-19 mitigations, including “lockdowns,” masking, and COVID-19 vaccine boosters, were always full of shit. After all, none of them could be so naive—at least, I don’t think they could be so naive, but have to admit the possibility—as not to have understood that RFK Jr. is not and never has been about rigorous science and that he certainly would never approve of Dr. Prasad’s “RCTs or STFU” attitude when it comes to his favored quackery and antivax pseudoscience. That they all so quickly capitulated and jettisoned their pandemic era approach to science they detested in favor of a Katzian “more fluid concept of evidence” leads me to the opinion that they were all either deluding themselves or lying all along. My only question now is: Are they useful idiots, opportunists, or both? You be the judge. (I vote for both.)
Finally, here’s one other tell, where Dr. Makary asserts, as he was in the middle of his little rant about levels of evidence being artificial and dogmatic:
We want to do things where evidence exists and want to pursue the creation of new evidence, but 60% of clinical decisions are purely discretionary where no evidence exists.
60% of clinical decisions have no evidence behind them? No evidence at all? While it’s true that a smaller fraction of clinical decisions have hard evidence in the form of rigorous RCTs behind them than we might like, I’ve never before seen an assertion that 60% of clinical decisions have no evidence behind them, not even from some of the biggest quacks I’ve ever encountered. Steve Novella took on the myth that most medicine is not based in rigorous science 18 years ago and tracked down a certain myth, namely that only 15% of clinical decisions are based on valid scientific evidence, and showed how it was a deceptive and almost certainly incorrect number and that more like 75% of medical interventions are based on compelling evidence, even if it’s not all RCT evidence. Basically, what Dr. Makary was doing was trying to claim that what most doctors do is not evidence-based in order to justify promoting RFK Jr.’s evidence-free stem cell quackery.
That’s because none of this is about “making America healthy again,” other than perhaps in the deluded minds of some MAHA true believers. Rather, it’s about making America safe for wellness influencers, health grifters, antivaxxers, and quacks. In this context, “health freedom” means freedom for quacks and charlatans to sell their wares unbothered by standards or law enforcement.


Dr. Vinay Prasad “Called For” RCTs. Dr. Peter Marks Delivered Them.
Dr. Vinay Prasad: “The establishment, the people who set the policies, they have to strive to get things right. They need to do studies. They need to be held to the highest standard.”
Jonathan Howard on December 19, 2025
Link
As one of the few who called for RCTs
Although COVID vaccines were brand new in February 2021, Dr. Vinay Prasad promised they would end the pandemic and berated people who took a medically conservative approach to them. He also praised pharma at the time, saying:
It’s really good news to see that covid cases are falling. Vaccine is amazingly effective & was developed so fast, I was impressed I’ve long been a critic of pharma, but they did a good job here. Good news.
He was right about this at least, and I still consider Operation Warp Speed to be the greatest scientific achievement of my lifetime. While countless people deserve credit for its success, one hero was Dr. Peter Marks, Dr. Prasad’s predecessor at the FDA. As early as April 2020, Dr. Marks formulated the plan that would lead to lifesaving vaccines before the end of the year.
While Dr. Marks was busy delivering RCTs, Dr. Prasad was busy glorifying himself for being “one of the few who called for RCTs” and castigating others for not doing the trials he “told them” to do in juvenile, profane rants. As I discussed previously, Dr. Prasad did this not to genuinely advance medical research, but rather to spread doubt and mistrust.
An evidence-based catastrophe of low quality evidence
Though this technique was manipulative and dishonest, it was undeniably effective. Today, Dr. Marks, who blasted Kennedy’s need for “subservient confirmation of his misinformation and lies” in his resignation letter, is no longer at the FDA. Dr. Prasad, however, is still weaponizing EBM to spread doubt and mistrust about vaccines. Nothing’s changed.
In his recently “leaked” memo, Dr. Prasad spoke of an “evidence-based catastrophe of low quality evidence” and demanded improvement. He wrote:
I want to outline a path forward. Our general approach in CBER will be to direct vaccine regulation towards evidence based medicine. This means: we will take swift action regarding this new safety concern, we will not be granting marketing authorization to vaccines in pregnant women based on unproven surrogate endpoints (any prior promises will be null and void), and we will demand premarket randomized trials assessing clinical endpoints for most new products. Pneumonia vaccine makers will have to show their products reduce pneumonia (at least in the post-market setting), and not merely generate antibody titers. Immunogenicity will no longer be used to expand indicated populations– these populations should be included in premarket RCTs.
We will revise the annual flu vaccine framework, which is an evidence-based catastrophe of low quality evidence, poor surrogate assays, and uncertain vaccine effectiveness measured in case-control studies with poor methods. We will reappraise safety and be honest in vaccine labels. I look forward to hearing your thoughts on how to do this better.
Additionally, at FDA, we have not been focused on understanding the benefits and harms of giving multiple vaccines at the same time. This is a concern shared by many Americans. The FDA’s standard has been to require randomized studies too small to draw any conclusions from– creating a false sense of efficacy and safety.
It would be unethical to use placebos and let people get pneumococcal pneumonia.
Of course, no one actively opposes better evidence, and Dr. Prasad is not unique in his desire for RCTs to answer every question about vaccines. Some of just also know that there are practical and ethical limitations to this approach. As the article FDA Official Proposes ‘Impossible’ Standards for Vaccine Testing That Could Curtail Access to Immunizations reported:
Demanding an RCT would prevent pregnant women from receiving most vaccines, said Jake Scott, MD, a clinical associate professor of infectious disease and geographic medicine at Stanford University School of Medicine.
That’s because pregnant women are almost never included in RCTs of vaccines or other drugs due to potential risks to the fetus, Scott said. Instead, the FDA does careful safety monitoring and measures antibody levels in women’s blood, which can show whether the vaccine is generating an immune response.
In his memo, Prasad also wrote that “pneumonia vaccine makers will have to show their products reduce pneumonia,” at least after they’re licensed, rather than provide indirect evidence of protection, such as antibody levels.
Prasad appears to refer to a childhood pneumonia vaccine that blocks infections with pneumococcus bacteria, which has been updated and improved several times over the years, said Dorit Reiss, PhD, a law professor at the University of California Law, San Francisco. Although the first version of the vaccine protected children from seven bacterial strains, the latest iteration protects against 20 strains.
Pneumococcal vaccines on the market today are approved based on antibody levels that correlate with protection and have been validated over decades, Scott said.
Proving that childhood vaccines prevent cases of pneumonia would be difficult, Scott said, because so many pathogens can cause pneumonia, including the flu, COVID-19 and respiratory syncytial virus (RSV).
Requiring manufacturers to conduct additional clinical trials every time they want to add additional viral or bacterial strains would delay updates by several years, all the while leaving people vulnerable to the bacteria, Reiss said. Some companies may decide that conducting such trials is too expensive.
“You’d need massive, expensive trials that many companies won’t pursue,” Scott said.
Conducting RCTs of proven vaccines could also be unethical, said Jesse Goodman, MD, who previously filled Prasad’s role at the FDA and now directs Georgetown University’s Center on Medical Product Access, Safety and Stewardship. “It would be unethical to use placebos and let people get pneumococcal pneumonia,” Goodman said.
Indeed, in a Prasadian universe, the results of RCTs would be available in 2027 to tell us exactly what we should have done in 2026. Pharma companies will be portrayed as villains for failing to conduct proper studies, and Dr. Prasad will portray himself as a heroic defender of EBM against barbarians who are indifferent to evidence. Meanwhile, pathogens will spread, and people will suffer preventable harms.
They need to do studies
It’s possible that Dr. Prasad’s critics are wrong about all this, and fresh RCTs are required for every tweak to every vaccine for every demographic subgroup. However, the techniques that catapulted Dr. Prasad to power aren’t sufficient now that he has power. He is the medical establishment now. As such, it’s not his job to merely “call for” RCTs- that’s our job as citizens. Rather it’s his job to deliver RCTs and to do so in a useful time frame via independent agencies that skeptical Americans will trust. Many people wouldn’t trust a study of the Pfizer vaccine paid for and carried out by Pfizer, and Dr. Prasad deliberately stoked mistrust of such research, writing:
If you run vaccine trials for Pfizer, the conflict doesn’t go away when the trial ends, because you’re going to run another one after you ram through their garbage products.
Fortunately, Dr. Prasad claimed to know exactly how to do these RCTs himself. Last year, along with Dr. John Ioannidis, he wrote an article titled Evidence Base for Yearly Respiratory Virus Vaccines: Current Status and Proposed Improved Strategies in which he said:
Well-conducted experimental studies, particularly randomized trials, are necessary to address persistent uncertainties about influenza and COVID-19 vaccines. We propose a new research framework which would render results relevant to the current or future respiratory viral seasons. We demonstrate that experimental studies are feasible by adopting a more pragmatic approach and provide strategies on how to do so. When it comes to implementing policies that seriously impact people’s lives, require substantial public resources and/or rely on widespread public acceptance, high evidence standards are desirable.
An entire section was titled “clinical trials are necessary and feasible.” He made it all sound so easy.
Even as a brand-new virus raged in 2020, Dr. Marks provided a template for Dr. Prasad on how intelligent, competent FDA leaders can produce concrete results for the American people. He tested and provided new vaccines in under a year. Now it’s Dr. Prasad’s turn to dazzle us all with his real-world prowess and abilities. As he put it when he was out of power and fluffing Kennedy:
The establishment, the people who set the policies, they have to strive to get things right. They need to do studies. They need to course correct. They have to admit their errors. They need to be held to the highest standard.
It’s hard to argue with that.
    • 

Lessons in confounding epidemiology: Household cleaning products, the microbiome and childhood obesity
Do eco-friendly cleaning products prevent obesity? Probably not, and you shouldn’t be eating them anyway.
Christopher Labos on March 14, 2019
Link

At first, you wouldn’t think that cleaning products, the microbiome of a young infant, and childhood obesity had much to do with each other. Of course, anything related to the microbiome is likely generate interest and news outlets will happily pick up any new study about it. A new study made quite a splash here in Canada, although I suspect it probably made less of an impact in the United States. The article, published in the Canadian Medical Association Journal, found that exposure to household disinfectants was associated with higher BMI in young children, possibly because it was altering their gut microbiome. There was also the suggestion that eco-friendly cleaning products protected against obesity.
As you can imagine, such an association certainly made for catchy headlines. The idea clearly builds on the concept of the hygiene hypothesis. For those who are not aware of it, the hygiene hypothesis is the idea that the introduction of antibiotics, and our cleaner environment overall, has contributed to rising rates of allergies and asthma. While there is some truth to the idea, the reality is more complex. What is often overlooked in this discussion is the fact allergic diseases started becoming more common in the later part of the 19th century, long before the development of antibiotics, and probably had more to do with people moving off of farms and into cities. Still it is true that avoiding potential allergens, like peanuts, has been shown to be counterintuitively counterproductive. In fact introducing peanuts early into the diet of high-risk infants can actually reduce the risk of developing a peanut allergy. We probably are better off being exposed to potential allergens early so that our immune system can better adapt to their presence.
So while not terribly likely, it is not impossible that antibiotics or disinfectants could have something to do with obesity by disrupting the balance our intestinal microbial flora. It is at least an idea worth studying. And while much has been written about the microbiome, it is important not to let our enthusiasm outstrip the science. Probiotics do have some uses, but they are not panaceas. But rather than getting into the broad concept of the hygiene hypothesis and the microbiome, which so many others have already written about, I wanted to look at this one specific study and explore a concept that is often brushed over very casually; the concept of confounding.
Confounding
Most people are aware of confounding in a general sense. It is the idea that a 3rd variable is affecting the interaction between an exposure and an outcome.Here, the causal relationships are indicated by the solid arrow, and the confounding relationship is indicated by the dashed arrow.
For example, people who carry lighters are at higher risk for developing lung cancer. This is true, as far as it goes, but it’s only true because people who carry lighters are also more likely to be smokers, and smokers are likely to get lung cancer. Thus, smoking confounds the association between carrying a lighter and developing lung cancer, i.e. it makes it seem as if lighters are related to lung cancer, but rather it is smoking that leads to both carrying a lighter and lung cancer.There are numerous other examples in the literature. Hormone replacement therapy (HRT) was thought to be protective for cardiac disease in women until the Women’s Health Initiative randomized trial showed that it was not. Likely the many observational trials on HRT simply reflected that the women taking hormone replacement therapy were more health conscious and therefore had a lower risk of heart disease at baseline.
The study
So let’s examine this new study in detail. This study was part of a larger cohort study called the Canadian Healthy Infant Longitudinal Development (CHILD) study, which is being used to examine a myriad of conditions. Women were recruited during their second or third trimester of pregnancy. At 3 months post-partum they filled out a questionnaire, which you can see here, about which types of cleaning products they used in their homes. Researchers grouped the answers into three categories: disinfectant, detergent and other. They then gave them scores for whether they used these products not at all (0 points), less than once a month (1 point), monthly (2 points), weekly (3 points) and daily (4 points). They then summed these scores to get an overall point score for each study subject.
At about 3 months of age, they also got a fecal sample from their babies to test their gut bacteria. At 1 year and 3 years of age, they assessed the children again to measure their height and weight.
So to summarize the flow of the study was:So what are the problems with this study? First, when you ask people to fill out a questionnaire, there is no guarantee that they will fill them out accurately. Apparently, researchers sent their research assistants into some homes to check whether the cleaning products that were reported in the questionnaires were actually present. They claim that there was high correlation between the self-reported questionnaire and the visual assessment of the research assistants, and cite a correlation coefficient r=0.32 (where zero means no correlation and 1 means perfect correlation). My experience has been that a correlation coefficient of 0.32 is not that great, and I wonder how accurately these questionnaires reflect the daily reality in the subject’s homes.
But setting that aside, this was not a randomized trial. Therefore we can easily imagine that the parents who used cleaning products regularly were probably different in some ways from the people who used them less regularly or not at all. In fact, the use of disinfectants was higher in one of the cities participating in the study (Edmonton) than it was in the other study sites. Also, disinfectant use was higher in households where infants were delivered by cesarean section, who received antibiotics during the delivery, and who were exposed to cigarette smoke in the home. Disinfectant use was lower in households were infants were breast-fed. In contrast, the eco-friendly products were used more often by mothers who breastfed their infants, and less often by women who were overweight pre-pregnancy.
So to summarize:
More disinfectant use	More eco-friendly product use
More caesarean deliveries	More breast feeding
More antibiotics during delivery	Mothers less overweight pre-pregnancy
More cigarette smoke in the home	
Less breastfeeding	
After analyzing the stool samples, researchers found that disinfectant use affected the gut microbiome, but detergents, eco friendly products, and other cleaning products did not. They principally saw an increase in Lachnospiraceae although there were differences in a few other bacterial species.
When it came to obesity outcomes at 3 years (the 1 year outcomes were not reported), children were more likely to be classified as overweight if their mothers were overweight before pregnancy, if they had a cesarean delivery, received antibiotics during delivery, were exposed to tobacco smoke in the home, and were not breast-fed.
So to summarize again:
Overweight in children at 3 years
Pre-pregnancy overweight in mothers
Cesarean delivery
Received antibiotics during delivery
Cigarette smoke in the home
Less breastfeeding
At this point, it should be clear what is going on. Households with more disinfectant use were more likely to have overweight children at age 3 years. But households with more disinfectant use were also more likely to have a slew of other factors that were likely contributing children’s weight gain and the changes in their microbiome.
Remember that stool samples were taken at 3 months of age. At this point, before children start eating solid food, one can assume that a child’s microbiome will be determined by a few important variables like whether or not they had a vaginal birth, whether or not they were breast fed, and whether or not they received antibiotics in their as-yet short life.
The interpretation
So one interpretation of these results is that disinfectant cleaning products lead to weight gain and that eco-friendly products protect against weight gain by making changes to gut bacteria. But another interpretation is that changes in the microbiome were being caused by other factors that happened to be associated with some cleaning products.There is no doubt that our environment plays a role in the rising rates of obesity. But I suspect much of that increase can be explained by our diet and our increasingly sedentary lifestyle. I am not certain if our cleaning products can explain that much of the current obesity problem in our society. I suspect that much of the effect size seen in this study can be explained by confounding, not causality.
With any non-randomized study confounding is a problem. It is incumbent on researchers to try to identify the variables that are tied to both exposure and outcome. And if researchers don’t do it, it’s up to you the reader to ask, “What else could affect this association?” We need to ask these questions because if we didn’t, we’d still be routinely giving women hormone replacement therapy.

   Why We Need Science: “I saw it with my own eyes” Is Not Enough 
Harriet Hall
Link
I recently wrote an article for a community newspaper attempting to explain to scientifically naive readers why testimonial “evidence” is unreliable; unfortunately, they decided not to print it. I considered using it here, but I thought it was too elementary for this audience. I have changed my mind and I am offering it below (with apologies to the majority of our readers), because it seems a few of our readers still don’t “get” why we have to use rigorous science to evaluate claims. People can be fooled, folks. All people. That includes me and it includes you. Richard Feynman said
The first principle is that you must not fool yourself--and you are the easiest person to fool.
Science is the only way to correct for our errors of perception and of attribution. It is the only way to make sure we are not fooling ourselves. Either Science-Based Medicine has not done a good job of explaining these vital facts, or some of our readers are unable or unwilling to understand our explanations.
Our commenters still frequently offer testimonials about how some CAM method “really worked for me.” They fail to understand that they have no basis for claiming that it “worked.” All they can really claim is that they observed an improvement following the treatment. That could indicate a real effect or it could indicate an inaccurate observation or it could indicate a post hoc ergo propter hoc error, a false assumption that temporal correlation meant causation. Such observations are only a starting point: we need to do science to find out what the observations mean.
Last week one of our commenters wrote the worst testimonial yet:
I have witnessed first hand the life that begins to flow through the body upon the removal of a subluxation.
What does this even mean? Does he expect anyone to believe this just because he says it? Would he believe me if I said I had witnessed first hand the invisible dragon in Carl Sagan’s garage?
Another commenter wrote
Those pro SBM commenters here seem to think that even if they see something with their own eyes that they can’t believe it if there are no double blinded officially published studies to prove that what they saw actually happened.
Well, yes, that’s pretty much what we do think; and we are appalled that you don’t understand it yet, since it’s the whole reason we have to do science. I would phrase it a bit differently: “Seeing something with my own eyes doesn’t prove it’s true and it doesn’t preclude the necessity for scientific testing.”
We can’t believe our own eyes. The very process of vision itself is an interpretive construct by the brain. There are two blind spots in our field of vision and we aren’t even aware of them. I saw a magician cut a woman in half on stage – that was an illusion, a false perception. I saw a patient get better after a treatment - my interpretation that the treatment caused the improvement may have been a mistake, a false attribution.
So for those who still don’t get it, here’s my simplistic article:
---------------------------------------
Sometimes We Get It Wrong
How can you know whether a medical treatment really works? If everybody says it works, and it worked for your Aunt Sally, and you try it and your symptoms go away, you can pretty well assume it really works. Right?
No, you can’t make that assumption, because sometimes we get it wrong. For many centuries doctors used leeches and lancets to relieve patients of their blood. They KNEW bloodletting worked. Everybody said it did. When you had a fever and the doctor bled you, you got better. Everyone knew of a friend or relative who had been at death’s door until bloodletting cured him. Doctors could recount thousands of successful cases.
All those people got it wrong. When George Washington got a bad throat infection, his doctors removed so much of his blood that his weakened body couldn’t recover, and he died. We finally got around to testing bloodletting and found out it did much more harm than good. Patients who got well had been getting well IN SPITE of bloodletting, not because of it. And some patients had died unnecessarily, like George Washington.
Even modern doctors sometimes get it wrong. Not long ago, doctors used to do an operation for heart disease where they opened the chest and tied off chest wall arteries to divert more blood flow to the heart. They had an impressive 90% success rate. A smart doctor named Leonard Cobb wanted to make sure, so he did an experiment where he just made the incision in the chest and closed it back up without actually doing anything. He discovered that just as many patients improved after the fake surgery! Doctors stopped doing that operation.
How could so many people be so wrong? How could they believe something had helped them when it actually had done them more harm than good? There are a whole lot of reasons people can come to believe an ineffective treatment works.
     1.  The disease may have run its natural course. A lot of diseases are self-limiting; the body’s natural healing processes restore people to health after a time. A cold usually goes away in a week or so. To find out if a cold remedy works, you have to keep records of successes and failures for a large enough number of patients to find out if they really get well faster with the remedy than without it.
     2.  Many diseases are cyclical. The symptoms of any disease fluctuate over time. We all know people with arthritis have bad days and good days. The pain gets worse for a while, then it gets better for a while. If you use a remedy when the pain is bad, it was probably about to start getting better anyway, so the remedy gets credit it doesn’t deserve.
     3.  We are all suggestible. If we’re told something is going to hurt, it’s more likely to hurt. If we’re told something is going to make it better, it probably will. We all know this: that’s why we kiss our children’s scrapes and bruises. Anything that distracts us from thinking about our symptoms is likely to help. In scientific studies that compare a real treatment to a placebo sugar pill, an average of 35% of people say they feel better after taking the sugar pill. The real treatment has to do better than that if we’re going to believe it’s really effective.
     4.  There may have been two treatments and the wrong one got the credit. If your doctor gave you a pill and you also took a home remedy, you may give the credit to the home remedy. Or maybe something else changed in your life at the same time that helped treat the illness and that was the real reason you got better.
     5.  The original diagnosis or prognosis may have been incorrect. Lots of people have supposedly been cured of cancer who never actually had cancer. Doctors who tell a patient he only has 6 months to live are only guessing and can guess wrong. The best they can do is say the average patient with that condition lives 6 months – but average means half of people live longer than that.
     6.  Temporary mood improvement can be confused with cure. If a practitioner makes you feel optimistic and hopeful, you may think you feel better when the disease is really unchanged.
     7.  Psychological needs can affect our behavior and our perceptions. When someone wants to believe badly enough, he can convince himself he has been helped. People have been known to deny the facts – to refuse to see that the tumor is still getting bigger. If they have invested time and money, they don’t want to admit it was wasted. We see what we want to see; we remember things the way we wish they had happened. When a doctor is sincerely trying to help a patient, the patient feels a sort of social obligation to please the doctor by improving.
     8.  We confuse correlation with causation. Just because an effect follows an action, that doesn’t necessarily mean the action caused the effect. When the rooster crows and then the sun comes up, we realize it’s not the crowing that made the sun come up. But when we take a pill and then we feel better, we assume it was the pill that made us feel better. We don’t stop to think that we might have felt better for some other reason. We jump to conclusions like the man who trained a flea to dance when it heard music, then cut the flea’s legs off one by one until it could no longer dance and concluded that the flea’s organ of hearing must be in its legs!
So there are lots of ways we can get it wrong. Luckily, there’s a way we can eventually get it right: by scientific testing. There’s nothing mysterious or complicated about science: it’s just a toolkit of common sense ways to test things. If you believe you’ve lost weight and you step on the scale to test your belief, that’s science. If you think you have a better way to grow carrots and you test your idea by planting two rows side by side, one with the old method and one with the new method, and see which row produces better carrots, that’s science. To test medicines, we can sort a large number of patients into two equal groups and give one group the treatment we’re testing and give the other group an inert placebo, like a sugar pill. If the group that got the active treatment does significantly better, the treatment probably really works.
Jacqueline Jones was a 50 year old woman who had suffered from asthma since the age of 2. She read about a miraculous herbal treatment that cured a host of ailments including asthma. She assumed the information was true, because it included lots of testimonials of people who had used it and were able to stop taking their asthma medications. They KNEW it worked. They had SEEN it work. Sick of the side-effects of conventional drugs, Jacqueline stopped using her three inhalers, steroids and nebulizer, and took the herbal supplement instead. Within two days she was in the hospital.
I had a massive asthma attack. I was seriously ill in hospital for six weeks, during which I developed pleurisy. I couldn't breathe and my lungs were so sensitive that even touching the area on the outside felt like someone was kicking me.
All those people who said that herbal remedy had cured their asthma got it wrong. Asthma symptoms fluctuate. Maybe their symptoms would have improved anyway. Whatever the reason, the remedy had not been tested scientifically and it was not effective for treating asthma, and believing those testimonials almost cost Jacqueline her life.
The next time a friend enthusiastically recommends a new treatment, stop and remember that they could be wrong. Remember Jacqueline Jones. Remember George Washington. Sometimes we get it wrong.

   The Role of Experience in Science-Based Medicine
Published by Harriet Hall
Link
Before we had EBM (evidence-based medicine) we had another kind of EBM: experience-based medicine. Mark Crislip has said that the three most dangerous words in medicine are “In my experience.” I agree wholeheartedly. On the other hand, it would be a mistake to discount experience entirely. Dynamite is dangerous too, but when handled with proper safety precautions it can be very useful in mining, road-building, and other endeavors.
When I was in med school, the professor would say “In my experience, drug A works better than drug B.” and we would take careful notes, follow his lead, and prescribe drug A unquestioningly. That is no longer acceptable. Today we ask for controlled studies that objectively compare drug A to drug B. That doesn’t mean the professor’s observations were entirely useless: experience, like anecdotes, can draw attention to things that are worth evaluating with the scientific method.
We don’t always have the pertinent scientific studies needed to make a clinical decision. When there is no hard evidence, a clinician’s experience may be all we have to go on. Knowing that a patient with disease X got better following treatment Y is a step above having no knowledge at all about X or Y. A small step, but arguably better than no step at all.
Experience is valuable in other ways. First, there’s the “been there, done that” phenomenon. Older doctors have seen more: they may recognize a diagnosis that less experienced doctors simply have never encountered. My dermatology professor in med school told us about a patient who had stumped him: she had an unusual dermatitis of her hands that was worst on her thumb and index finger. His father, also a doctor, asked her if she had geraniums at home. She did. She had been plucking off the dead leaves and was reacting to a chemical in the leaves. The older doctor had seen it before; his son hadn’t.
Then there’s what we loosely call “intuition.” It can be misleading, but it can also be a function of pattern recognition that has not risen to the level of conscious awareness. Experience can help us perceive that “something just isn’t right” about a patient or a working diagnosis. An experienced doctor may get a feeling that a patient might have a certain disease. He couldn’t justify his hunch to another doctor, but he has subconsciously recognized a constellation of findings that were present in other patients he has seen. Of course, he would still need to do appropriate tests to confirm the diagnosis, but he might do more tests and do them sooner than a less experienced doctor. This kind of pattern recognition has been called the “Aunt Tillie” phenomenon: you can spot your Aunt Tillie’s face in a crowd, but you couldn’t tell someone else how to do it. You just know Aunt Tillie when you see her. Computer face recognition is learning how to do this, but it uses measurements, not the gestalt method our brains use.
Then there’s the wisdom that (sometimes) comes with age. I’ve just been reading Marc Agronin’s book How We Age where he shows that old age is not all bad. As we get older, we are not able to accomplish mental tasks as fast, and our short-term memory declines; but there are compensations. We are more able to integrate thinking and feeling, less likely to get carried away by emotions, better able to see both sides of an issue, and better able to cope with ambiguity. We can develop more patience, acceptance, tolerance, and pragmatism in dealing with complex situations. We have a vast store of life experiences to bring to the table, helping us put things into a more realistic perspective. Wisdom is elusive: not every elder develops it. I’m sure you can all think of many counterexamples.
Medicine is an applied science, and the same science can be applied in different ways by different doctors. There are times when two science-based doctors can look at the same body of evidence and still disagree about what it really means or about what to do for a specific patient. There is room for disagreement and for different approaches. Scientific medicine is often criticized for focusing on the disease rather than on the person who has the disease. I have known patients who have turned to alternative providers because of a bad experience with a science-based doctor’s poor communication skills or “bedside manner.” We can aspire to a kinder, gentler, more personal science-based medicine where experience and improving people skills are integrated with science (a kind of “integrative medicine” that actually makes sense.)
It’s not clear whether you are better off with a young doctor or an older one. A young doctor is more likely to be up to date on the latest science; an older doctor might make better patient-centered decisions. A younger doctor might be better at tuning up your bodily vehicle; an older one might be better at helping you decide when to drive it, where to go, and how fast. A young doctor might offer the latest treatment; an older one might question whether it is really preferable to an older treatment for that particular individual, or even question whether any treatment is really necessary at all.
Conclusion:
In summary, while “in my experience” claims can be dangerous, experience does have a role to play in science-based medicine.
Disclaimer:
As an ORF (Old Retired… something) and a Medicare-card-carrying senior citizen, I am biased. I have a vested interest in thinking that I have improved with age and experience. This is an opinion piece and I can’t cite any controlled studies to support my opinions. I’m almost tempted to insert tongue firmly into cheek and say “Trust me; I’m a doctor.”

I Never Meta Analysis I Really Like
Mark Crislip on October 5, 2012
Link
David Gorski recently pointed out that Science Based Medicine is going on five years. Amazing. That there would be so much to write about day after day comes as a surprise to me. Somehow I vaguely thought that ‘controversies’ would be resolved. Pick a SCAM, contrast the SCAM with reality as best we understand it, and, once the SCAM was found wanting, it would be abandoned. Why would rational, thoughtful people persist in the pursuit of irrational behavior, contradicted by the universe?
Ha. More the fool me. I would never have guessed that these SCAMs are harder to kill than Dracula (at least one version of Dracula). Stake them and back they come*.
I have tried to avoid repeating repeating information found in prior posts by myself and others, in part because I am lazy and in part because, well, I have said it before. Just look it up. I have come to realize (all too slowly) that each blog entry should be self contained and that much of the old material is lost in the corn maze (an punning homophone) that is WordPress. Reading my second favorite computer reinforces the realization that each post often needs to be an island universe, complete in itself.
Medicine is difficult. At least I find it difficult. You have to first come up with a cause of the symptoms, understand the underlying physiology and then try and determine the best course of therapy. I have it easy in Infectious Diseases, a much more binary specialty. Patients are either infected or not, I have a therapy or not and I either cure them or not. Not a lot of wiggle room in the treatment of, say endocarditis or pneumonia, and very little in the way of bias and placebo to obfuscate the therapy. Or so I would think.
Much of the rest of medicine is not as clear cut. As a physician there are multiple ways to assess the potential efficacy of a therapy. One is a definitive randomized placebo controlled trial. Those do not seem to occur as often this decade as in prior times when giants walked the earth. Or so my faulty memory suggests.
Another way is to try and master the literature, a futile endeavor. In two areas of medicine I have more than passing knowledge: Infectious Diseases and SCAM’s. The nice thing about having a breadth and depth of reading is you can understand where new articles fit into an overall picture. What was the patient population studied, what were the weaknesses of the study, how applicable is the information to other populations? The downside of such expertise is I have to rely on the kindness of strangers in other areas of medicine, since I know next to nothing.
The breadth of knowledge has also made sources of information that I once trusted much more suspect. Popular media? No way. Newspapers and magazines so often get it wrong in areas of my expertise that I no longer trust them in areas where I know nothing (just about everything else).
Meta-analysis? Nope. Talk about disillusion. I used to rely on meta-analysis, but they are worse than laws and sausages, ceasing to inspire respect in proportion as we know how they are made. I still like the meta-analysis and systemic reviews as a nice overview of a clinical topic, but, for reasons we will see, I am hesitant to draw any therapeutic conclusions from any meta-analysis.
And the worst source is an anecdote. If I have said it once, I have said it once, the three most dangerous words in medicine are “in my experience.” I had always though of it in the context of physicians deciding on a therapeutic intervention, not so much from the patients perspective. I realized that for patients that is often the primary way they deciding to try a therapy, especially a SCAM therapy.
What would you do if you were a highly intelligent intellectual at one of the top newspapers and were curious about acupuncture? With the research capacity of, say, the Chicago Sun Times, you could get the skinny on any topic, right? Or instead, you could ask you readers if you should get acupuncture. Sigh. A microcosm of why, compared to SBM bloggers, Sisyphus had it easy.
That method of seeking medical advice has been around since 480 AC as mentioned by Herodotos:
‘They had a very practical habit: on the marketplace passer-by’s giveadvice to the patient about their ailment, in which case they can sometimes rely on personal experience or take advantage of someone else who suffered from the same symptoms. Nobody can just pass by without saying anything. It is obligatory to ask the patient what ails him. They bury their corpses in honey and mourn in the same way as in Egypt.’ (Thanks to Cees Renckens for the quote).
The more things change, the more they stay the same. Except for the honey part.
Everything I distrust about medical reporting seems, of late, to have found a home in the Atlantic. Check out the headline: Biological Implausibility Aside, Acupuncture Works by Lindsay Abrams.
Really? News to me. I tend towrds the opinion that if a process is biologically impossible, then it is more likely that a) any positive results are likely to be spurious and due to bias and b) if bias is removed, the old double blind placebo controlled trial, then most effects will fade to zero.
It often appears that journals and magazines do not even bother to read their own reports. A scant two years ago the Atlantic reported on the work of John Ioannidis and why most of the medical literature is suspect
“Much of what medical researchers conclude in their studies is misleading, exaggerated, or flat-out wrong. So why are doctors—to a striking extent—still drawing upon misinformation in their everyday practice?”
and that in clinical trials
“the range of errors being committed was astonishing: from what questions researchers posed, to how they set up the studies, to which patients they recruited for the studies, to which measurements they took, to how they analyzed the data, to how they presented their results, to how particular studies came to be published in medical journals.
This array suggested a bigger, underlying dysfunction, and Ioannidis thought he knew what it was. “The studies were biased,” he says. “Sometimes they were overtly biased. Sometimes it was difficult to see the bias, but it was there.” “
One of the epiphanies of my understanding in how to approach the medical literature was when I discovered the works of Dr. Ioannidis. It would appear that Lindsay Abrams didn’t read the article about Dr. Ioannidis. More likely it was read and ignored. All the issues raised about the validity of the results of medical research never seem to translate into SCAM studies.
What was this earth shattering, high quality clinical paper that demonstrated after decades of lousy research that acupuncture works? Finally a large randomized placebo controlled clinical trial with good endpoints that demonstrates efficacy? Certainly all the prior trials that meet those criteria demonstrate acupuncture doesn’t work and that acupuncture is no better than placebo.
Readers of my entries know that my assessment of placebo: it does not alter pathophysiology, just the perception of pain and disease, and then only a wee, clinically irrelevant, amount. I agree with the article that acupuncture is “nothing more than fancy ways of invoking the placebo.” And you know Crislip’s law: acupuncture effect = placebo and placebo effect = nothing, therefore the acupuncture effect = nothing.
I well know that interacting with a patient with a therapeutic intervention will alter the patients perception of what is occurring without altering the underlying pathophysiology. As the NEJM study revealed, if patients think they are receiving a therapy even when they are not, they will report their asthma is better even when the objective tests show no improvement. To my mind the placebo effect is no different than kissing a child’s boo-boo: it is subjectively beneficial although no objective changes occur to the boo-boo. These human interactions are an important, if ineffective, part of the human relationships.
The Atlantic was referring to
“new, large study out of Memorial Sloan-Kettering Cancer Center, published in the Archives of Internal Medicine, cautiously suggests that there is indeed something more to acupuncture. A meta-analysis of 18,000 patients from 29 randomized controlled studies, it found that the treatment was more effective than controls in relieving back and neck pain, osteoarthritis, chronic headache, and shoulder pain. Significantly, it also found that real acupuncture was more effective than shams.”
Sigh. I see the word study, and I think oh good. Someone has done a clinical trial. Enrolled patients and compared, in a double blind manner, interventions against placebo. No. It is a meta-analysis. Calling a meta-analysis a study is not unlike a library declaring they acquired several dozen new books and magazines when their copy of Readers Digest arrives. Someone massaging preexisting data; hardly a study. Nice for an overview of a topic, but worthless for drawing conclusions definitive conclusions.
When applied to real treatments, those based on reality and known physiology, the results of meta-analysis are often not predictive of well done clinical trials:
The outcomes of the 12 large randomized, controlled trials that we studied were not predicted accurately 35 percent of the time by the meta-analyses published previously on the same topics.
Of course, that presupposes that the topic is not tooth fairy science. If the interaction is based on nonsense, then the validity of an intervention is likely to be less. My bias, clearly stated here, is that one should be able to be able to reason up and down the pathophysiologic pathways of disease and treatment on basic biologic plausibility and basic principals. It comes from being a truly holistic doctor, understanding infections at the level of amino acid substitution leading to drug resistance or disease susceptibility through to the interactions with the earths ecosystem.
That meta-analyses are unreliable should not be surprising. Most reality based clinical trials are flawed, and if you collect a series of flawed studies, you end up with one big flaw. The idea behind the meta-analysis, collecting all the cow pies into one big pile and making gold, is only as good as the quality of the initial pies. It is rare to collect Marie Callender’s.
I am not a true skeptic. I tend to avoid the words implausible and unlikely where I do not think they apply. As best can be determined, acupuncture, homeopathy, energy therapies, and a host of other SCAMs are, at the level of first principals, total bunk. And legitimate therapies cannot flow from pure bunk nor can a meta-analysis make pumpkin from cow.
Prior meta-analyses of acupuncture suggest they are substandard. The perhaps ironic and self referential Systematic Review of Systematic Reviews of Acupuncture published 1996-2005 suggests
“Systematic reviews of acupuncture have overstated effectiveness by including studies likely to be biased. They provide no robust evidence that acupuncture works for any indication.”
And as the Wikipedia points out
“The most severe weakness and abuse of meta-analysis often occurs when the person or persons doing the meta-analysis have an economic, social, or political agenda… If a meta-analysis is conducted by an individual or organization with a bias or predetermined desired outcome, it should be treated as highly suspect or having a high likelihood of being “junk science.” From an integrity perspective, researchers with a bias should avoid meta-analysis and use a less abuse-prone (or independent) form of research. “
This is no small issue if you happen to be a practicing clinician who takes the health and wealth of your patients seriously. Do I trust a meta-analysis that suggests linezolid is better than vancomycin for the treatment of pneumonia when the studies and the meta-analysis are sponsored by the company? It doesn’t completely abrogate the results of the trials, but it is well known that when the researcher has an axe to grind, the results will tend to sharpen the axe of the researcher. All studies have bias, the question is how well they compensate. I usually discount the results by about half when applied to the real world.
When approaching any study where the end points are subjective and at the limits of perception, two archetype need to be considered. The first is the endless ability of people to fool themselves.
At the turn of the last century a French Physicist, Blondlot, discovered N-rays. These rays were at the limit of detection and, like acupuncture, made no sense in the context of know reality. Multiple papers were published on N rays until a visiting professor, unknown to Blondlot, incapacitated the machine yet Blondlot still saw the N-rays.
N-rays were a purely subjective phenomenon, with the scientists involved having recorded data that matched their expectations.
Beware the N rays.
The other archetype is clever Hans, the counting horse, who actually was reading the nonverbal cues from his owner to know when to stop counting. Humans are probably more sensitive and skilled than horses at reading nonverbal cues, leading to
The observer-expectancy effect (also called the experimenter-expectancy effect,expectancy bias, observer effect, or experimenter effect) is a form of reactivity in which searcher’s cognitive bias causes them to unconsciously influence the participants of an experiment. It is a significant threat to a study’s internal validity, and is therefore typically controlled using a double-blind experimental design.
Both N-rays and clever Hans are examples of the importance of blinding the researcher and the patient and why if blinding is not adequate then any results are suspect, especially if the end points are subjective.
The studies of acupuncture that exclude N-ray and Clever Hans effects (i.e. really, really blinded) always show that acupuncture does nothing compared to fake acupuncture.
The classic interpretation of clinical trials when an intervention, say mammary artery ligation is no better than a sham mammary artery ligation, then neither are effective. I have been in medicine a long time and that career is littered with interventions that were discovered to be no better than sham/placebo and (sometime slowly and reluctantly) abandoned by the medical profession. That, of course, has yet to happen with any of the SCAMs discussed on this blog. While no hospital still offers mammary artery ligation for angina, Centers of Integrative Medicine and their ilk become more common.
So that is the background I use in approaching a clinical trial, a meta-analysis, and a paper on SCAMs. I admit I have to read meta-analysis with Mr Gumby like understanding of the manipulation of the data, since statistics and I have never got along, despite my affinity for other forms of mathematics.
So how does “Acupuncture for Chronic Pain Individual Patient Data Meta-analysis” stand up?
Color me unimpressed.
Done by Acupuncture Trialists’ Collaboration. So no different than a meta-analysis by big pharma; they have a dog in the fight. That’s ok, we all have a dog in the fight. But any result is likely inflated.
After a search 31 of 82 trials met their criteria for inclusion. Or they left out 51 trials, the validity of which I cannot say. Would any of that information changed the outcomes of the analysis? I don’t know; I lack the time to look at all the primary data in preparation of this entry.
They found that sham and real (as if there is a difference) acupuncutre were more effective than doing nothing. I would expect that. The boo-boo is being tended too; there will be salubrious results.
Did they remove the N-ray and Clever Hans bias? Nope.
“health care providers obviously were aware of the treatment provided, and, as such, a certain degree of bias of our effect estimate for specific effects cannot be entirely ruled out.”
Right there you know that the results are mostly worthless:
“Putting their results into context, the authors of the study explain that for a pain rating of 60 on a 100-point scale, follow-up scores decreased to around 43 for those had received no treatment, 35 for those who had received fake treatment, and 30 for those who received acupuncture. This translates into a 50 percent reduction in pain for the acupuncture patients, and only 30 and 42.5 percent reductions for the control and placebo groups, respectively.
It is impossible to measure pain objectively (Radiolab did a great piece on this last week), and the difference in pain reduction between sham and true acupuncture, though statistically significant, was small. But the authors’ methodical elimination of biases, coupled with their massive sample size, give weight to their findings.
The biases are there probably more enough to account for the results of real being better than sham acupuncure, the effect inconsequential clinically, and a massive sample size of cow pies only leads to one big cow pie indeed.
Dr. Strawman then weighs in:
Should the lack of biological plausibility lead us to reject compassion and empathy as a means to help improve our patients’ health?
How about
Should the lack of biological plausibility lead us to reject a costly and worthless therapy with known complications and instead use our compassion and empathy in a manner that is not based on lies?
The Fail blog needs to link to the Atlantic. It is a perfect match.
That is not all that is new about acupuncture of late. The Cochrane reviews continues to prove they have no standards or they are really getting bored. They did a review of mumps treated with acupuncture. Really. They found one study. Honestly. They meta-analyzed away:
We could not reach any confident conclusions about the efficacy and safety of acupuncture based on one study. More high-quality research is needed.
They had the same conclusion with insomnia and endometriosis.
Of course, pain, mumps, insomnia and endometriosis all have the same underlying pathophysiology, no wonder it ‘works’ for so many diseases. Somehow I don’t think we need to do any research on using acupuncture to treat mumps, much less do a review.
Not unsurpizingly when there is double blinding, analgesic effects disappears. This was an interesting study since they treated acute pain rather than chronic pain patients who
gather a lot of information about different pain treatments and firmly believe in different therapies.
They tested the ability of acupuncture, sham or real, to effect acute pain (cold and capsaicin). It didn’t. While the real acupuncuture worked slightly better than sham for capsaisen-induced pain, the effect, like a Vickers results, “occurred mainly in a rating range that seemed irrelevant to clinical pain.”
In another study in the same issue of Pain there is no difference between sham (random punctures), ‘real’ or placebo (fake punctures) acupuncture for low back pain, but all three are better than conventional therapy. The big flaw in the study is that they used hardened acupuncturists with a mean of 8.5 years of practice who were not blinded. While not statically different, real performed better than sham better than placebo, suggesting a clever Hans effect.
Like the asthma study in the NEJM, the best one can conclude is that acupuncture is that it is another beer goggles of alternative medicine: it convinces the wearer the disease looks better than it actually is.
* I am avoiding a Whack-a-mole metaphor.

Cochrane Reviews: The Food Babe of Medicine?
Mark Crislip on May 2, 2014
Link
There are two topics about which I know a fair amount. The first is Infectious Disease. I am expert in ID, Board Certified and certified bored, by the ABIM. The other, although to a lesser extent, is SCAMs.
When I read the literature on these topics, I do so with extensive knowledge and, in the case of ID, 30 years of clinical experience. The extensive knowledge, and, one hopes, understanding, has led me to read meta-analyses with a grain of salt substitute. They average meta-analysis and systematic review is good for gaining a general understanding of the topic within, as well as, and here is the key phrase, the limitations of the included studies.
And like all the published literature, when writing a meta-analysis, those with an axe to grind will grind it. Even, or perhaps especially, the Cochrane reviews.
Just because something is labelled as a systematic review does not mean it is any good. We have to be just as vigilant now as ever. Even a review with a Cochrane label does not make its true. Four out of 12 Cochrane reviews on acupuncture were wrong. Caveat lector rules, OK?
Randomized placebo-controlled trials can be definitive if done well. But all too often they are done on a well-defined population and how widely applicable the results are, is always a question. When I was a resident it seemed all the cardiovascular intervention studies were done on a VA population and we wondered how applicable outcomes on old, white, male, smokers were to other populations.
All clinical trials are part of a context of the medical literature and Xigris was a classic example of how a single study may not be definitive or even right. Even in a field as binary as ID, it is not always clear cut what the right intervention is from individual clinical trials and you have to try and synthesize a conclusion from the preponderance of the data.
As I have mentioned before, the theory behind meta-analyses is that if you gather all the cow pies into one big pile you get clinical gold. I think they are often better as an overview on a topic rather than as a tool for coming to a conclusion as to the benefit of a particular intervention, especially since meta-analysis do a poor job of predicting the results of a (perhaps) definitive (ha) clinical trial.
GIGO is perhaps the more common result of a meta-analysis. Reading the Cochrane review of late, I am less certain that their output is gold, but a larger pile of the original material.
Take oseltamivir (aka Tamiflu). Please.
As mentioned, I am an Infectious Disease doctor and this year was the second-worst influenza outbreak of my career. One of the hospitals in our system is a level-one trauma center and we offer ECMO, so we see those who are the most ill from influenza. I will say, on balance, I was glad we have oseltamivir during outbreaks.
How (or perhaps do) ID docs think? Infections are caused by germs. Duh. But not everyone thinks so. Germs come in a wide variety of forms: viruses, bacteria, fungi, parasites and more. I once did a rough count and, if you are splitter, there are about 1,200 common germs I need to know to do my job. See. ID isn’t that hard.
We treat these germs with antibiotics. Antibiotics usually interfere with one biochemical pathway or another to kill or disable the germ. Antiseptics, like alcohol, kill non-specifically. Some antibiotics are cidal (they kill germs) and some are static (they stop the germ from functioning, but once the antibiotics is removed they get back to business as usual). For many infections, but not all, static is good enough. Stop the bug from dividing and the host immune system, if there is one, will have time to catch up and take care of the infection.
Viruses are not alive (the debate as to what is alive can go on in comments), so they can’t be killed. It is part of reason antivirals are just OK as a therapeutic intervention. It would be nice if our drugs against viruses were as good as our drugs against S. pyogenes, eradicating them. But they don’t.
As a clinician what you expect with an antiviral is to take the edge off the illness. If you have genital herpes, you will get fewer attacks with less severity. For HIV it took the use of three antivirals to have a significant effect of the virus and they are still not a cure. And for influenza you would expect a mild amelioration of the disease given that the drugs are usually given late into a process of robust viral replication.
Of course not all viruses are the same and not all infections are the same. Outcomes depend on a complex integration of pathogen life cycle, pathogenicity, the host’s immune function and the ability of the medication to affect the organism. And sometimes, for some patients, the difference between doing well and doing poorly may be due to relatively small shifts in favor of the infection.
Oseltamivir is an OK drug for the treatment of influenza. By the time most patients get to a doctor, get the medication filled and then get steady state with drug, the virus has had several days to run rampant and I would not expect that oseltamivir would, in normal people, do much more than take the edge off. And, as the recent meta-analysis in the BMJ demonstrates, that is what oseltamivir does in relatively normal people in the outpatient setting.
Oseltamivir for influenza in adults and children: systematic review of clinical study reports and summary of regulatory comments
As we have known for a while, oseltamivir shortens illness by about a day. In relatively normal outpatients. It may reduce pneumonia and it modestly reduces the risk of symptomatic flu in individuals and households. And there are side effects. There always are. The results of the meta-analysis confirm what was known. Oseltamivir is a drug of modest efficacy in treating influenza in a mostly normal outpatient population.
The authors did a tremendous amount of work and the basic results are sound. Then the editorializing slips in, ground to the fineness of the atomic axe.
Two of the authors have a strong bias against the treatment and prevention of influenza by the current methodologies. I have discussed Jefferson’s semi-coherent arguments concerning influenza in another post and it is safe to say that Doshi and I are not on the same planet about influenza since he considers that:
For the vast majority, influenza is unpleasant but self-limiting.
It is a characteristic of those who are against the current treatments and prevention of infections to directly, or indirectly, minimize those who suffer and die from infections. They usually use the word ‘only’ in front of morbidity and mortality statistics.
It is certainly true that I see the minority of influenza patients: those trying to die. The 18,000 who died in 2009 in the US of H1N1 were a minority of the 60.8 million cases. As a physician I tend to side with John Donne in my approach to the ill and dying:
No man is an island,Entire of itself,Every man is a piece of the continent,A part of the main.If a clod be washed away by the sea,Europe is the less.As well as if a promontory were.As well as if a manor of thy friend’sOr of thine own were:Any man’s death diminishes me,Because I am involved in mankind,And therefore never send to know for whom the bell tolls;It tolls for thee.
If the patient is sick enough to be hospitalized, prompt therapy with oseltamivir probably decreases the odds of needing ICU care and if you are in the ICU or pregnant, probably decreases your chances of dying. Note “decreases the chance”. The data is not great, but consistent: in hospitalized patients, oseltamivir decreases the chance of death. When thinking about treating populations during a pandemic, having sufficient medication available will be a good thing for those ill enough to require hospitalization.
The problem if you are deciding to stockpile drug for an infection like influenza is virus’s variability. This century we have had influenza with high mortality but low infectivity and low mortality and high infectivity. When, not if, we get a strain of influenza with both high infectivity and mortality, we will probably be thankful that we have oseltamivir and given the data it will probably prevent some from dying of the disease.
Of course, those in public health are screwed no matter what they do. If they prepare and the pandemic does not occur, they have wasted money. And if there is an event that is the ID-equivalent of Katrina, no preparation will ever be enough. The scariest thing in my career was the 2009 H1N1, when in Portland every bed in the ICU and every ventilator we had was being used. If another patient came in with flu and needed ICU support we had nothing to offer them. And, by total luck, we peaked exactly at our surge capacity. And, thanks to the ICU, ECMO and maybe oseltamivir, some people that should have died, did not.
Whether we should stockpile anti-influenza medications, spend the money and time, is an interesting political questions. Of course we never have honest conversations about health care in the US, so let’s make a decision based on a good study with flawed and misleading propaganda in the discussion and conclusion. Sounds about right for a county that got all wiggens about death panels.
However, I do not think the data on the efficacy for treating influenza in relatively healthy outpatient populations during influenza seasons of relatively low mortality really informs decisions as to the appropriateness of stockpiling drugs for a repeat of the pandemic of 1919 (an estimated 675,000 Americans among the dead) or if one of the bird influenzas becomes more contagious.
They allude to this exact issue in their conclusion:
Given that oseltamivir is now recommended as an essential medicine for the treatment of seriously ill patients or those in higher risk groups with pandemic influenza, the issues of mode of action, lack of sizeable benefits, and toxicity are of concern. This is made worse by the record and stated intentions of governments to distribute oseltamivir to healthy people to prevent complications and interrupt transmission on the basis of a published evidence base that has been affected by reporting bias, ghost authorship, and poor methods.
We believe these findings provide reason to question the stockpiling of oseltamivir, its inclusion on the WHO list of essential drugs, and its use in clinical practice as an anti-influenza drug.
Note: seriously ill patients or those in higher risk groups. Those not covered in this meta-analysis. Those who may die and, at least from observational studies, get benefit from oseltamivir. I like how they analyzes one patient population then tries to apply it to another population where the preponderance of data suggests benefit. And if you are dying of flu, those side effects are of a little less importance. I would take a bit of a headache and kidney dysfunction to avoid being buried 6 feet under.
Not that it stopped anyone from reporting the conclusions of the study without a modicum of critical thought. Most of the secondary reporting repeated the no stockpiling message and that oseltamivir therapy wasn’t good. The anti-Tamiflu propaganda component seems to have far exceeded the actual results of the paper.
I would lean towards the stockpile. I like to keep the bell from tolling. But to use the BMJ as evidence against stockpiling for a more virulent pandemic is, to my mind, inappropriate. Like I have often said, a meta-analysis is useful if you can use it to confirm your pre-existing bias and ignored if it does not.
That bias can lead to some really Food Babian discussion, as is evident in the BMJ article. They suggest in the conclusion that there are
the issues of mode of action
for oseltamivir. The paper says
Oseltamivir may have anti-inflammatory properties that make people with influenza-like illness feel better by shortening the duration of symptoms and reducing the occurrence of symptoms
and
Sufficient plasma concentrations of oseltamivir carboxylate from orally administered oseltamivir phosphate may act directly on the host’s endogenous neuraminidase to reduce (or suppress) the immune response. The potential hypothermic or antipyretic effect of free oseltamivir as a central nervous system depressant may also contribute to the apparent reduction of symptoms.
As Mr. Doo says, “Huh?” Both of these paragraphs are not referenced. Residents at my hospitals know I am enamored of the immunomodulatory effects of antibiotics. I had never heard of either of these effects with oseltamivir so I went looking.
In mice, oseltamivir given at 30 to 1,000 times the human dose causes hypothermia. In another mouse model oseltamivir led to a decrease in a variety of pulmonary cytokines. Color me unimpressed, and if you are going to use the mouse model, might as well include all the animal studies that show increased survival with oseltamivir. No wait. Only including statements that support your contention is the Food Babian way.
Oseltamivir is a neuraminidase inhibitor, serving as a competitive inhibitor of the activity of the viral neuraminidase (NA) enzyme upon sialic acid, found on glycoproteins on the surface of normal host cells. By blocking the activity of the enzyme, oseltamivir prevents new viral particles from being released by infected cells.
As a rule when you inhibit viral replication you give the immune system a chance to catch up and you slow down the disease. I can find no data that the immunomodulatory effects of oseltamivir are relevant in human disease at standard dosing and the data in humans and animals all point to the usual mechanism by which an antimicrobial works.
Not only is the press not bothering to read the article with more critical thinking than Food Babe, the Wikipedia editors fell for it hook, line and stinker [sic].
Oseltamivir might induce a low immune response with low levels of pro-inflammatory cytokines. This may reduce symptoms of influenza, but is not related to inhibition of virus replication. There is also a potential temperature lowering effect that can contribute to symptom reduction. The influenza virus specific mechanism of action proposed by the producers does not fit the clinical evidence. Evidence rather suggests a multi-system and central action.
Not. It is amazing how easily and rapidly cow pies get spread on the internet. I suppose a meta-analysis gets halfway around the web before the truth has a chance to get its shoes on.
As best I call tell, these two ideas are more for the purpose of FUD. It is of Food Babian proportions to suggest the primary mode of an antiviral is due to immunomodulatory effects instead of messing with viral replication, considering the nature of the data supporting the idea. All the data suggests benefit by slowing down the virus.
They continue
Oseltamivir relieves symptoms in otherwise healthy children but has no effect on children with asthma who have influenza-like illness, a population that should most benefit from its intake.
Should it? I don’t know. That is why we do studies. I would expect valacyclovir suppression to help with recurrent herpes meningitis. It doesn’t. It makes the disease worse. That doesn’t mean that valacyclovir is ineffective for other manifestations of the disease or in other patient populations.
An explanation for this finding is in the nature of the young asthmatic population, which is well cared for and used to a regular intake of powerful drugs and close follow-up. The incremental benefit of oseltamivir assumption is thus likely to be undetectable in such a population.
On the other hand, once lung inflammation started because influenza I would expect cough and SOB to be hard to control as the asthma kicked in, and inhaled steroids, known to increase the risk of pneumonia and TB, may suppress local inflammation to decrease the local immune response in favor of the infection and prolong illness.
The oseltamivir trials did not detect any influenza related mortality events, a reflection of the benign nature of influenza and influenza-like illness and perhaps trial design.
Or it could have been healthier populations with prior immunity from prior disease exposure and vaccinations. Outpatient antibiotic studies are usually done in patients with little risk for mortality and until the 2009 H1N1 flu seasons have been relatively mild, except to those who died. The mortality from flu leading up to the release of oseltamivir in 1996 were comparatively low. Test a drug during a benign flu season and you will not see mortality.
There was a lot of good information in the article, but the bias and FUD rendered it, like much of the Cochrane influenza reviews, more infomercial than a nuanced discussion of a complicated topic. And while I didn’t go through all the supplementary material, I did not see that they suggested oseltamivir is used in the manufacture of yoga mats. But not for a lack of trying.
There are other goofy Cochrane reviews that leave we wondering about their standards. They evidently operate on the assumption that every intervention, no matter how ridiculous, should be subject to a meta-analysis. When all you have is a hammer, everything is a nail. So we get acupuncture for mumps.
We could not reach any confident conclusions about the efficacy and safety of acupuncture based on one study. More high-quality research is needed.
No it isn’t.
Or Homeopathic Oscillococcinum(®) for preventing and treating influenza and influenza-like illness..
Our findings do not rule out the possibility that Oscillococcinum(®) could have a clinically useful treatment effect [although reality does ed.] but, given the low quality of the eligible studies, the evidence is not compelling.
They think a meta-analysis on magic is going to have compelling results. It is a tough job but someone has to see if magic is effective. But if you want the ultimate in goofy, you have to turn to “Vitamin C for preventing and treating the common cold” which contains what may be the most dumbass statement in the medical literature of 2014. Here goes:
Authors’ conclusions .The failure of vitamin C supplementation to reduce the incidence of colds in the general population indicates that routine vitamin C supplementation is not justified, yet vitamin C may be useful for people exposed to brief periods of severe physical exercise. Regular supplementation trials have shown that vitamin C reduces the duration of colds, but this was not replicated in the few therapeutic trials that have been carried out.
Fine so far. The literature shows vitamin C is of minimal benefit, and probably none, for cold treatment and prevention. As the main conclusions say
Thirty one comparisons examined the effect of regular vitamin C on common cold duration (9745 episodes). In adults the duration of colds was reduced by 8% (3% to 12%) and in children by 14% (7% to 21%). In children, 1 to 2 g/day vitamin C shortened colds by 18%. The severity of colds was also reduced by regular vitamin C administration.Seven comparisons examined the effect of therapeutic vitamin C (3249 episodes). No consistent effect of vitamin C was seen on the duration or severity of colds in the therapeutic trials.
Here it comes. Ready? It is really muttonheaded. Do not say I did not warn you, because, as my favorite box of blinking lights likes to say, it has napalm levels of burning stupid.
Nevertheless, given the consistent effect of vitamin C on the duration and severity of colds in the regular supplementation studies, and the low cost and safety,
Here it comes, the apex of idiocy.
it may be worthwhile for common cold patients to test on an individual basis whether therapeutic vitamin C is beneficial for them.
A more pathetic example of what’s-the-harm shrugery I cannot imagine.
As if there is any way a patient can really know if vitamin C does anything. Cochrane, please, give me a framework I can suggest to my patients that they can apply to help determine if vitamin C is effective. I am waiting. Godot will arrive sooner.
The uselessness of personal experience in determining efficacy of medical interventions is why we do clinical trials. For crying out loud, I though it the raison d’être of the whole Cochrane Collaborative: relying on evidence instead of anecdotes. Wrong.
The biases alone render such an endeavor useless. It is why the words “In my experience” are the most dangerous in medicine for determining if an intervention is effective. And it is what they suggest. Why even bother with the analysis in the first place?
If there is a worse statement in all the published evidenced-based medicine, let me know.
There was a time when I was under the impression that the conclusions of a Cochrane review were of value. No more. Now I see one and a roll my eyes. What kind of Food Babian nonsense is this one going to have?

Diagnostic Reflections
At the end of a long and storied career I reflect upon making a medical diagnosis. No reflection is one way to diagnose a vampire.
Mark Crislip on January 26, 2023
Link

When I was younger, I could remember anything, whether it had happened or not; but my faculties are decaying now and soon I shall be so I cannot remember any but the things that never happened. It is sad to go to pieces like this but we all have to do it.
—Mark Twain
There was a recent study that raised a mild brouhaha. Titled “Diagnostic Errors in the Emergency Department: A Systematic Review“, it found that there was often a misdiagnosis made in the ER. And there was gambling, gambling! in Ricks.
The study found 15 medical problems that accounted for the bulk of the misdiagnosis:
The top 15 clinical conditions associated with serious misdiagnosis-related harms (accounting for 68% [95% CI 66 to 71] of serious harms) were (1) stroke, (2) myocardial infarction, (3) aortic aneurysm and dissection, (4) spinal cord compression and injury, (5) venous thromboembolism, (6/7 – tie) meningitis and encephalitis, (6/7 – tie) sepsis, (8) lung cancer, (9) traumatic brain injury and traumatic intracranial hemorrhage, (10) arterial thromboembolism, (11) spinal and intracranial abscess, (12) cardiac arrhythmia, (13) pneumonia, (14) gastrointestinal perforation and rupture, and (15) intestinal obstruction.
They estimated:
With 130 million U.S. ED visits, estimated rates for diagnostic error (5.7%), misdiagnosis-related harms (2.0%), and serious misdiagnosis-related harms (0.3%) could translate to more than 7 million errors, 2.5 million harms, and 350,000 patients suffering potentially preventable permanent disability or death.
So, not an insignificant problem.
My first thought was to contrast that finding with the practice of fantasy-based medicine, those who follow the nonsensical ravings of a lunatic mind (NRLM). You know, alternative medicine.
Practitioners of Traditional Chinese-Pseudo Medicine are, after examining the tongue and pulse, 100% wrong in their diagnosis. Chiropractors? As long as they are finding and treating subluxations, 100% wrong. Naturopaths? There I am less certain, as that is a form of NRLM that excels more in goofy treatment than in goofy diagnosis. Most of the other forms of NRLM hover close to 100%. The alterative medicine diagnosis is definitional for misdiagnosis.
But that’s not a valid comparison, that of apples and birdshit, as Sal the Cacophony might call it. Making a diagnosis is, like Teen Talk Barbie’s opinion on math, tough.
As this post goes up, I have one working day left as an Infectious Disease doctor. If you include the second two years of medical school, I have spent 41 years working mostly in the hospital taking care of acutely ill people. I have easily seen 50,000 patients in consultation. Really. 4-5 consults a day, 48 weeks a year, 1 in three to 1 in 5 weekends on call. It adds up to a small city worth of patients.
As I head into my inevitable decline, like all old geezers, I want to reflect on the past and pontificate to all you youngsters who will likely ignore my sage opinions.
Getting the correct diagnosis in the fog of medicine can be damn difficult and, like Grampa Simpson, I will take this opportunity to give a mostly SBM free rambling on medical diagnosis and my experiences, both failures and triumphs.
For I flatter myself that I am a diagnostician. Making the correct diagnosis is key to all that follows in patient care and, as I mentioned, tough. Anyone can look up the correct treatment once a diagnosis is made. Although I remain shocked at often that simple google is not done. Hence, a career.
As another aside, Google Scholar can be invaluable in making the right diagnosis. Has influenza been associated with sterile cardiac vegetations? Yep. Can vomiting lead to esophageal tear and empyema? You betcha. Two examples from this month where the diagnosis of atypical diseases presentations was explained by a quick google search. I am so glad the Index Medicus has gone the way of the dial up modem. Now if we could just get rid of medical references behind paywalls.
Making the diagnosis is what makes the real doctor. Petty as it is, I derive enormous satisfaction from reaching a diagnosis missed by others, especially if the diagnosis is not an infectious disease. Brings out the gloat like nothing else.
For example, I had a patient who came home to Portland to die of AIDS. This was the bad old days before HAART. He had been at Brigham and Women’s Hospital in Boston for two months slowly fading away. When I saw him? Tanned like he had been in Jamaica in August instead of a Boston winter. Low blood pressure and sodium, high potassium. Classic Addison’s, induced by his rifampin. Hard to miss. A bit of steroids and he perked up, living another year. Gloat central to this day.
As Gag Halfrunt noted, “Vell, Brigham and Women’s is just zis hospital, you know?” But I learned as a resident in Minnesota that the so called great hospitals? Meh. Patients often went to Rochester for a second opinion only to come back with no new insights but significantly poorer. Hold the Mayo, we used to say.
When I am asked about a second opinion, I always think of the Henny Youngman joke: My doctor told me I was fat. I said I wanted a second opinion. He said, “OK, you’re ugly, too”.
Not all doctors are diagnosticians. What is a diagnostician? Someone who, after doing a history and physical, comes up with the diagnosis for what ever ails the patient. I remember once I had an AIDS patient, a home brewer, who had Saccharomyces cerevisiae in his blood. So I knew what aled him. Yep. I went there.
Pathologists and radiologists do not qualify as diagnosticians. Nor do many other physicians. Some are fine within their organ/specialty. A cardiologist is fine for a cardiac diagnosis. A gastroenterologist is fine for a GI diagnosis. Larry is fine for a Three Stooges diagnosis.
ER docs? Most assuredly not diagnosticians. Sorry guys. The usual attitude of physicians at the receiving end of an admission from the ER is that the ER is wrong about the diagnosis. But that is how it should be. Part of the ER docs job is to determine whose illness needs admission. Every doctor has their specific training and expertise. Just as you would not want me to remove your appendix or deliver a baby, you would not expect all doctors to be equal diagnosticians.
If you want a doctor who has both breadth and depth of medical knowledge, you want either an internal medicine/hospitalist or an ID doctor. I, like Zaphod Beeblebrox, would survive the total perspective vortex.
As an aside. Kids today. I remain shocked – shocked! – at how few house staff or medical students have any knowledge of the classics. Monty Python and The Hitchhiker’s Guide to the Galaxy are unknown to them. Doctors are not the dorky nerds we used to be.
Infectious diseases involve all organ systems and many processes can mimic an infection. As my old boss used to say, the ID doc has to be the second-best cardiologist, the second-best pulmonologist, etc. in the hospital.
So what does it take to be a diagnostician? Besides overweening self-esteem?
Here is where the old geezer wanders down memory lane with an onion on my belt, as was the style of the time.
Experience helps. While I have long maintained the three most dangerous words in medicine are in my experience, that has more to do with choosing a treatment than making a diagnosis. The practice of medicine, with emphasis on practice, is as much a trade as a profession, and like building cabinets, the more patients you have seen, the better diagnostician you become. Unlike, say, golf, where no amount of practice will allow you to escape mediocracy. Sigh. That is life’s motto: Practice makes mediocre.
A couple of years ago I introduced myself to a patient who apparently was admitted from central casting as a stereotypical Russian. A bear of a man, he got out of bed and gave to a crushing hug.
“Finally,” he said. “An old doctor. Someone who knows what they are doing.”
Это точно.
While I have not seen it all, I have seen more than most. It is seeing diseases that are important. You can read about a disease, but it never really gives the flavor of a patient presentation.
The textbooks will say 89% of patients will have this symptom and 67% that symptom, and 5% will have the other thing. But patients do not read the textbooks and so often present with signs and symptoms kind of sort of resembling the textbook description, but not quite. Like Nutrimatic tea.
Years ago I saw a patient in clinic on Friday afternoon. A bad time to be sick. And she was sick. High fevers, severe polyarticular arthralgias, rash, toxic appearing. It had been going on for a week. I had no clue what she had, but she was sick. I was betting on endocarditis with immune complex arthritis and admitted her.
Monday, I came back to work and the patient was gone. I asked the hospitalist what happened, fearing the worst.
“Oh. It was Still’s disease. Got better on steroids and we sent her home. Classic.”
Classic? I had vague memories of Still’s and read up on it. Yep. Classic case. Totally missed it. But I learned about Still’s and vowed not to miss a case again.
Wrong. Three months later I was consulted on another case, not classic, and missed the diagnosis again. Rheumatology made the diagnosis on the basis of a sky-high ferritin. Since that time? I have not missed a case and have made the diagnosis several times. If you have never seen a case, you will not likely recognize it the first time. Or the second time. There is no better experience for becoming good at diagnosing a disease than missing a few cases. From the patient’s perspective, you do not want to be the first for some doctors learning experience. There is a saying in medicine, you don’t want to be a great case.
When you see a patient, we are trained to generate a differential diagnosis: a list of diseases that can cause the patients symptoms. The 10 reasons for chest pain. The 12 reasons for shortness of breath. The 4 causes of fever of unknown origin. I actually didn’t bother to count them up, so do not quibble in the comments.
The weird thing about that process of generating lists is how, over time, it has become internalized. It used to be that I would consciously go through a list for whatever the problem was, considering the pros and cons of each option. Now? More often than not, I know the diagnosis and with remarkably little thought and scant information. The answer just burbles up like a bubble in a tar pit: pop. It’s leptospirosis. It is really weird how much processing goes on beneath my geriatric consciousness. And it impresses other docs no end when you give the diagnosis after just a few words.
I like to swing for the fence that way. Progressive headache for two months and an abnormal LP? Cryptococcus. Pancytopenia and fevers for a month in a Hispanic? Brucella. And Bingo was his name-o. It is important to remember that while Babe Ruth led the league in home runs, he also led in strikeouts. People always remember the former and forget the latter. You can make quite a name for yourself swinging for the fences and being right only occasionally.
Pulling information seemingly out of thin air impresses patients as well. Years ago I saw a young female with, well, a relapsing fever. So rather than the usual ‘any travel lately’ question, I asked, “How was your vacation in Black Butte?” The look of shock when she asked how I knew she had been to Black Butte was priceless. I understand how mentalists must feel. But Black Butte is the only part of Oregon where Borrelia, the spirochete that causes relapsing fever, is found. I went to the lab and there was the wee beastie on a smear.
I still mentally go through the list. Gotta be safe. And I often make sure I consider four lists: the list of what the diagnosis might be. The list of what diagnosis would be potentially catastrophic to miss. The list of cool diagnoses. Of course, I am unnaturally drawn to the cool diagnosis, no matter how remote the possibility, and it is painful to not go looking for diseases that I know have almost zero prior plausibility of being the cause of the patient’s symptoms. And lastly, depending on the case, I might mentally run through organism class: could it be viral, rickettsia, spirochete, bacteria, fungal, parasite, etc.? I once counted about 1,400 bugs I need to know, or at least know of, to be an ID doc. More than Pokémon (1,008), but I still gotta catch ’em all.
There is another way to categorize and consider disease presentation for diagnosis:
    • The common presentation of a common disease.
    • The uncommon presentation of a common disease.
    • The common presentation of an uncommon disease.
    • And the worst: an uncommon presentation of an uncommon disease.
    • And even worser: an uncommon disease presenting like a common disease.
I was not surprised to see that epidural abscesses were on the miss list in the ER. It is an uncommon disease that often presents like a common disease. Many of the cases I see have had a delay in diagnosis because it starts as persisting back pain after minor trauma. No signs of infection. It is not until the disease progresses to some sort of paralysis that the MRI is ordered. We can’t do every test on every patient every time. There is not the time, resources, or money.
The fog of war has nothing on the fog of diagnosis.
Medicine is the realm of uncertainty; three quarters of the factors on which action in medicine is based are wrapped in a fog of greater or lesser uncertainty. A sensitive and discriminating judgment is called for; a skilled intelligence to scent out the truth.
You know. An ID doctor. But it is not unusual when initially seeing a patient to not have a clue as to what is going on. Lymphoma? Endocarditis? Still’s again? Often early in a disease the presentation is too vague to point to a specific disease. I am looking for a pattern and there is no pattern I can see. What then? That is a common problem in the ER.
Early in my career it was don’t just stand there, do something. Now it’s don’t just do something, stand there. So often time will clarify what the diagnosis might be and what needs to be done. As Laozi noted, and I often quote, “Let muddy water stand and it will become clear.” It is important to explain to the patient why you are doing nothing, as they often want something, anything, done. I often compare the problem to unripe fruit; it takes time for the disease to mature to the point it can be recognized. The ER is filled with green bananas that are yellow by the time I am called to see the patient.
ID docs do have a reputation for doing excessively detailed histories to aid in making a diagnosis and for writing extremely long notes.
Another aside, if you have not discovered Dr. Glaucomfleken, you are in for a real treat. I have never met the man, but he has evidently followed me on a hike.
Producing War and Peace for each consult is a real fetish that I do not understand. Many cases are simple, especially if you know what you are doing. By the way, if you really want an experienced diagnostician, avoid a University Hospital. Think about it. Often the attending is on service 3 months a year. So they might have 25% of the experience of someone who works every day in the trenches. And it is often indirect experience: patient to medical student to resident to fellow to attending and back, the perfect opportunity for medical telephone.
My time in skepticism and SBM has made me a far better physician and diagnostician. Skeptics are, I wager, more aware of how we think and the errors in thought that lead to faulty diagnosis:
The most common cognitive problems involved faulty synthesis. Premature closure, i.e. the failure to continue considering reasonable alternatives after an initial diagnosis was reached, was the single most common cause. Other common causes included faulty context generation, misjudging the salience of findings, faulty perception, and errors arising from the use of heuristics. Faulty or inadequate knowledge was uncommon.
Because of my time at SBM, I have spent an inordinate amount of time thinking about thinking. As a result I think I think better. One of those thought processes is to consciously consider the outliers. Often all the patients presenting signs and symptoms can be accounted for with one diagnosis, or at least we try to. Occam, not Hickam, rules. Newbies and the unexperienced focus on the what is consistent with the presumptive diagnosis. I make a point of considering the inconsistencies, occasionally resulting in a different diagnosis. Like the aforementioned empyema. Huge pleural infection with an oral Streptococcus but almost no lung infection. There was no pneumonia to cause the empyema. It was thought the empyema was due to aspiration, but careful questioning found the aspiration event occurred during an episode of prolonged and violent vomiting. Boerhaave’s syndrome was a better explanation. Focusing on the outliers is a learned habit that does not come naturally.
There are two, and likely more, patterns in medicine that I think will adversely affect the future development of good diagnosticians.
One is a lack of continuity. Everyone is a shift worker. A week on and a week off. In my hospitals I am the only doc that is there day in and day out, except for vacation and some weekends. I suspect physicians are not getting the understanding of disease that only comes with following a patient from admit to discharge and then outpatient follow-up. Continuity of care is important to gain understanding of disease and is the best opportunity to see your, well, screwups. I always tell residents it is OK to make a mistake once, it is how you learn. Nothing imprints a disease like a mistake. Just don’t do it twice. Except in diagnosing Still’s disease.
The other is EPIC aka the electronic medical record. The way rounds works now is everyone sees their patients then vanishes to the computer room to chart. I rarely see other docs, attendings or residents, as I wander the halls. The constant interaction with other providers has vanished and with it the cross-fertilization that occurs with discussing the odd cases. I used to get multiple questions a day from docs I would run into. Now? Maybe one a day. On a busy day.
I do not know what the solution is to improving diagnostic acumen. The comment section of the NYT article suggests doctors should listen to patients better. I doubt that. Patients don’t give histories, they tell stories. Digression-filled stories from which we need to extract the pattern of the patient’s disease. It is amazing how much of what I need to know (symptoms, onset, progression, current status) seems at times to be almost deliberately avoided by patients in favor of details I do not need or information I have ready access to in the chart. Time, concentration, stamina, empathy are all limited resources in a busy day. I understand why doctors interrupt the patient around 11 seconds in. Often they are not telling you anything you need to know to figure out the problem at hand.
Given that:
For a given disease, nonspecific or atypical symptoms increase the likelihood of error
And:
These issues are not unique to the ED—they are seen across clinical settings, regardless of study method.
I am not certain if there is any way to make physicians better diagnosticians. Most of the solutions I find online seem a bit, well, trite. They come down to thinking better. Which is hard to do. Or teach. Just look at the cognitive bias codex. It is a wonder that humans can force themselves to think clearly. Rational thought, I wonder, is perhaps at its core an unnatural act. And there are those who just seem unable to process the information to come up with the right diagnosis. 1+1= cheese. Always odd to see. But how to counter it?
One of the sobering experiences of my career is coming across the charts of patients I took care of long ago. Reading my notes I wrote as an intern when I was a third-year resident or coming across my notes from early in my practice.
Man. As Mark Twain famously sort of noted:
When I was an intern, I was so ignorant I could hardly be called a doctor. But when I got to be an attending, I was astonished at how much I had learned in seven years.
My current skill set is a result of 40 years of hospital clinical medicine and all that comes with it. I do not think there is a shortcut to all the lessons learned. Unfortunately. I expect some misdiagnosis will always be part of the human condition.
Self aggrandizement
If you should have in interest in the diagnosis and treatment of infectious diseases cases, there is no better source than my Puswhisperer series, six and counting. Harriet liked them.

Comprehending the Incomprehensible
Mark Crislip on March 22, 2013
Link
Medicine is impossible. Really. The amount of information that flows out the interwebs is amazing and the time to absorb it is comparatively tiny.
If you work, sleep and have a family, once those responsibilities are complete there is remarkably little time to keep up with the primary literature. I have made two of my hobbies (blogging and podcasting) dovetail with my professional need to keep up to date, but most health care providers lack the DSM-4 diagnoses to consistently keep up.
So we all rely on short cuts. People rely on me to put new infectious disease information into context and there are those I rely upon to help me understand information both in my specialty and in fields that are unrelated to ID.
Up and down the medical hierarchies we trust that others are doing their best to understand the too numerous to count aspects of medicine that no single person could ever comprehend.
If I want to know about the state of the art on the treatment of atypical mycobacterium or how best to treat Waldenströms or who knows the most about diagnosing sarcoid, there is always someone who can distill their expertise on a topic to the benefit of the patient and my knowledge.
Trusting others is the biggest shortcut we routinely take in medicine to wade through the Brogdignagian amounts of information that flood into medical practice. We have to trust other clinicians, the researchers and the journals that all the information is gathered and interpreted honestly and accurately.
I understand that the world is a tricky and confusing place and that even under the best of circumstances the literature has ample opportunity to be wrong. But in the end the truth, or some approximation of it, will out.
Trust is a fragile foundation upon which to build an edifice, but the practice of medicine would be impossible without it. It is one of the reasons medical fraud is particularly heinous; it strikes to the heart of the practice of medicine.
One of the other shortcuts we use is statistics. It is a quick and dirty way to check the validity of results and there is nothing like a good p-value to make a result believable. The smaller the p-value, the better. That is a simple, and sometimes misleading, approach. Except for ID and SCAM’s, I rarely have the luxury of time to read a study closely, so I look at the p-value instead.
I have long had a mental block with statistics. I took, and dropped, statistics once a year for 4 years in college. Once they got past the bell shaped curve and the coin flip they would lose me. So I have to trust that the statistics are correct when I read a paper, and that bothers me. I would feel better if I knew that at one time I had been able to crank out the results with a pencil and a piece of paper.
Otherwise statistics are like the old New Yorker cartoon.
It was nice to be reminded for the umpteenth time that statistics can be tricky, with an article in Vaccine called 5 ways statistics can fool you—Tips for practicing clinicians. I write this, and my other blog, to first educate and entertain myself. As a side effect I hope to educate and entertain others, but I long ago realized that not everyone shares my aesthetic. There are those of you reading this for which the 5 ways are old hat, part of your critical thinking skills. For me it is yet another attempt to understand statistical concepts with the depth that I have with MRSA, rather than, say, the loop of Henle, the understanding of the latter lasts as long as I am reading about it. If that. I still suspect the loop has as much validity as homeopathy.
Their opening statement is a master of understatement:
However, compounding the problem of finding and effectively using the medical literature is the fact that many, if not most, physicians lack core skills in epidemiology and statistics to allow them to properly and efficiently evaluate new research. This may limit their abilities to provide the best evidence-based care to patients.
No kidding. And this article is meant to be applied to reality/science based treatment. As I mentioned in my last blog entry, it is even more problematic when statistics are applied to fantasy interventions like acupuncture or homeopathy. Then, not only do most physicians lack the core competencies to evaluate the paper, most are not able to recognize the subtle biases that allow magic to perceived as real.
It is like having real scientists evaluate ESP, where a magician would be a better qualified observer. I suspect that many editors and reviewers of SCAM papers are untrained in the skills required to evaluate SCAM research. They apply the rules of science where those rules do not apply.
The tips, with examples from the vaccine literature, are
Tip #1: statistical significance does not equate to clinical significance
Tip#2: absolute risk rather than relative risk informs clinical significance
Tip#3: confidence intervals offer more information than p-values
Tip#4: beware multiple testing and the isolated significant p-value
Tip #5: absence of evidence is not evidence of absence
It is remarkable how these tips can be applied to SCAM-related articles and the articles are found wanting. For example, the acupuncture article I reviewed last week: barely significant p-values, (not true, thats why the line through), large confidence intervals and misleading multiple tests; the sine qua non of positive SCAM studies.
This, of course, is applying statistics to reality. Add a touch of prior plausibility and there is no reason to suspect that the barely positive effects noted in some SCAM studies are due to anything but bias and only bias. Since most of the interventions we discuss in this blog have zero probability of a real effect, there should be an alternative explanation for what appear to be beneficial outcomes. In the case of SCAM, absence of evidence is very likely evidence of absence.
Recently there were two other articles that looked at the effect of bias on outcomes in clinical trials. Observer bias in randomised clinical trials with binary outcomes: systematic review of trials with both blinded and non-blinded outcome assessors. and Observer bias in randomized clinical trials with measurement scale outcomes: a systematic review of trials with both blinded and nonblinded assessors
Almost the same article twice over.
In the two analyses they looked at the outcomes of clinical trials where the person determining the outcome, for example, the number of wrinkles after a therapy, are blinded or not as to the therapy. No surprise, when people know what the intervention is they assessed it as more effective than when they were blinded.
Nonblinded assessors of subjective measurement scale outcomes in randomized clinical trials tended to generate substantially biased effect sizes.
And this is for relatively tightly controlled clinical trials. In real-world practice the tendency to over estimate the effect of a therapy must be even greater. It shows up most often in the phrase “I use X on my patients and that is how I know it works.”
The strong effect of bias in determining that you see what you want to see, regardless of what is actually there, is a common human characteristic. To be blind to that characteristic is a common Dunning-Kruger variant of all true believers.
Statistics makes my brain hurt. But if I remember some basic principals it makes understanding the medical literature a bit more comprehensible.

Do doctors pay attention to negative randomized clinical trials?
David Gorski on September 22, 2014
Link

We at the Science-Based Medicine blog believe that all medicine, regardless of where it comes from, should be held to a single science-based standard with regards to efficacy, effectiveness, and safety. We tend to focus primarily on “complementary and alternative medicine” (CAM), now more commonly known as “integrative medicine,” because (1) we believe it to be undermining the scientific basis of medicine and allowing outright quackery (or, as I like to refer to it, quackademic medicine) to infiltrate medical academia, which is fast becoming medical quackademia and (2) because when it comes to the other threats to the scientific basis of medicine as it is practiced today, such as pharma influence and various medical dogmatism, there are a whole host of active critics better equipped and more energetic than we are who can do the job (usually) better. That is why, whenever I hear advocates of CAM/”integrative medicine” attack us for not spending enough time on various corruptions of clinical trial processes or the perfidy of big pharma, I tend to gently tell them in my characteristically diplomatic manner that that’s what I like to call the “Why don’t you blog about what I think is important and interesting instead of what you think is important and interesting?” criticism, then I refer them to our posts on John Ioannidis, overdiagnosis and overtreatment, the shortcomings of mammography, or any number of other posts we at SBM have done through the years pointing out where current medical practice falls short. Indeed, it never fails to amuse me to point out how angry an eminent radiologist became at me for my posts criticizing him for his misleading attacks on certain studies that question the value of screening mammography.
In particular, though, I like to point out a post I did on vertebroplasty as a treatment for vertebral compression fractures (VCFs) due to osteoporosis. Basically, I discussed then recent evidence showing how vertebroplasty for such fractures is, basically, placebo medicine, no better than acupuncture. Indeed, I likened the state of evidence regarding vertebroplasty to that of acupuncture, in which small, pilot studies appear to be positive, but then the follow-up rigorous randomized clinical trials fail to find a benefit greater than that of placebo. It turns out that a rather telling study regarding vertebroplasty was published earlier this year that I somehow missed that addresses a problem we have in “conventional” medicine.
I became aware of this new study when I was perusing the latest issue of the Journal of the American College of Surgeons (JACS), hot off the presses. (OK, in reality it wasn’t the latest issue. It was the September issue; I’m more than a month behind in my perusing of the medical literature.) This study, however, had been published online back in April. It must not have gotten much news coverage then, but it’s an important study, because it is one of the few studies out there that look explicitly at the effect of randomized clinical trials with negative results. The study is by Fabrice Smieliauskas, PhD, Sandi Lam, MD, MBA, and David H. Howard, PhD, and is entitled “Impact of Negative Clinical Trial Results for Vertebroplasty on Vertebral Augmentation Procedure Rates.” Smieliauskas et al. set the stage by describing what vertebroplasty is. Basically, it’s a procedure that’s done by either spine surgeons or radiologists in which bone cement (polymethelmethacrylate, or PMMA) is injected into the fracture site under radiologic guidance. The intent is to stabilize the fracture and relieve the pain. Kyphoplasty is a variant of vertebroplasty in which a device, such as a balloon, is inflated or expanded in the vertebral body. In theory, this restores vertebral body height and creates a cavity for the PMMA to enter. These procedures are considered similar enough that they are often referred to as “vertebral augmentation procedures” (VAPS).
Now here’s the real issue:
In August 2009, the New England Journal of Medicine published 2 randomized controlled trials testing vertebroplasty vs sham surgery for patients with osteoporotic VCFs. Both found that patients randomized to vertebroplasty did not experience decreases in pain or disability relative to patients in the placebo arm. In these studies, both groups showed improvement over time, suggesting a natural history for reduced pain and/or placebo response. The trials were covered in the lay press in newspapers, internet articles, and television news reports. In September 2010, the American Academy of Orthopaedic Surgeons (AAOS) issued a guideline that advised against the use of vertebroplasty for patients with osteoporotic VCFs. The guideline listed kyphoplasty as a treatment option but noted that evidence to support its use is “limited.” Many payers have restricted coverage of VAPs to patients whose symptoms have failed to resolve after a course of optimal medical therapy, but these types of coverage restrictions can be difficult to enforce in practice.
These are the studies that I blogged about five years ago, as well as four years prior to that for my not-so-super-secret other blog, where I used the whole question of VAPS as an example of a scientific delusion in reference to earlier studies of VAPS and the anecdotal evidence used to justify it, which I characterized as testimonials not unlike alternative medicine testimonials. Of course, given that randomized clinical trials (RCTs) are the “gold standard,” what’s of most interest is whether practice changes after major negative RCTs, like the two negative RCTs (linked to here and here). I discussed them in detail at the time. Suffice to say, taken together, they were pretty definitive evidence that VAPS for osteoporotic vertebral compression fractures do not work. Besides having been published in a journal with as high a visibility as the NEJM, these two RCTs got a lot of publicity in the lay press, including a major article in the New York Times. The question that remained to be answered is whether physicians changed their practice in response to two resoundingly negative clinical trials that capped off a series of smaller studies questioning the value of VAPS.
The answer is: Yes and no, at least in Florida. Also, as they say in Facebook status updates, it’s complicated, as you will see.
So what did Smieliauskas et al. do? They examined quarterly trends in VAPS use using the Florida hospital inpatient discharge and ambulatory surgery databases from 2005 to 2012, which capture a 100% sample of surgical spine procedures performed in the state in hospitals, hospital surgery centers, and freestanding surgery centers. They chose Florida because it is large and sociodemographically diverse. Discharge data was supplemented with data on physician specialty from the Florida Practitioner Profile, and population counts were obtained from the 2000 and 2010 Censuses and US population counts from the 2012 Census Population Estimates. This allowed the investigators to adjust for population growth during that period. So, yes, one weakness of this study is that it only covers Florida, and, as already mentioned, there is a huge regional variation in the use of VAPS throughout the US. The investigators also cautioned that it’s possible that some of the decline in some procedures was due to the recession due to the financial meltdown that occurred in 2008, but if that’s true, then these results would look even worse.
First up, we see that the average age of patients was 76 years, with Medicare being the primary payer in 85% of cases. Procedures were done by interventional radiologists (27.9%); orthopedic surgeons (32.2%); neurosurgeons (22.3%); other specialties (7.1%) and “unidentified” (10.6%). Interventional radiologists performed 57% of vertebroplasties and 18% of kyphoplasties, while orthopedic surgeons performed 13% of vertebroplasties and 38% of kyphoplasties. A little more than half of the vertebroplasties were performed on an outpatient basis, while most kyphoplasties were performed in an inpatient setting.
Here’s the money figure. The TRIALS line indicates quarter before publication of vertebroplasty trials in the NEJM. The AAOS line indicates quarter before release of the American Academy of Orthopaedic Surgeons statement on vertebral augmentation procedures:
Looking at the graph, it looks as though VAPS rates did decline after the studies. Indeed, as the authors reported:
The interrupted time series estimates of the effect of the trials generally confirm the observations in the figures (Table 2). Overall, the vertebroplasty rate declined by 51.5% (column 1) and the kyphoplasty rate declined by 40.0% (column 2). Both results were statistically significant (p < 0.010). Similar declines (53.8% and 38.9%; columns 3 and 4, respectively) were observed for these procedures when restricting the sample to patients with osteoporotic fractures only. The change in the vertebroplasty rate was driven by a time trend effect, while the change in kyphoplasties was driven by a level effect and a time trend effect.
The interesting observation really comes when the investigators break down declines in VAPS by specialty:
The striking differences by physician specialty were confirmed in the regression results. Interventional radiologists responded to the trials (column 7) with an increase in time trend of the vertebroplasty rate (+0.019, p = 0.046) and a percentage change after the trials that was not significantly different from 0. This was very different (p < 0.010) from physicians in other specialties, whose vertebroplasty rate declined by 73.1% after the trials (column 8). Interestingly, radiologists exhibited a greater decline in the kyphoplasty rate (−46.9%) than did other specialists (−21.7%, p < 0.010) (columns 9 and 10).
Remember, according to this database, interventional radiologists do far more vertebroplasties than they did kyphoplasties, while surgeons do more kyphoplasties than vertebroplasties. Also, kyphoplasty is viewed as a more “invasive” procedure than vertebroplasty, mainly because of the inflation of the balloon in the vertebral body. So it might be that radiologists were more willing to shy away from kyphoplasty, leaving that to surgeons, than they were vertebroplasty. Moreover, surgeons, viewing the “more invasive” kyphoplasty procedure as more “surgical” in nature, might be less willing to give it up, given that the two studies didn’t directly address kyphoplasty. However, it is disturbing that the rates of vertebroplasty done by interventional radiologists haven’t declined in Florida at all since these 2009 studies, while among other specialties have decreased their use of vertebroplasty markedly.
So, yes, as I’ve described before, physicians really do abandon the use of procedures that are shown to be ineffective in large randomized clinical trials, but the process is messy and often arguably takes longer than it should. One other aspect of this process is that specialty society recommendations matter. Note that it took only a year for the AAOS to issue its guideline that strongly advised against the use of vertebroplasty for patients with osteoporotic VCFs. And so vertebroplasty use among specialties other than interventional radiology (the vast majority of whom were orthopedic surgeons or neurosurgeons) declined by nearly three-quarters. However, the guideline noted that evidence to support the use of kyphoplasty is “limited” but still retained it as an option for the treatment of osteoporotic VCFs. As a result, the decline in use of kyphoplasty was much more modest among surgeons. The authors note that:
Perhaps not coincidentally, the specialty societies’ practice guidelines after the trials interpreted the clinical trial evidence differently. The AAOS guidelines strongly recommended against vertebroplasty for osteoporotic VCFs, implying that physicians should follow the recommendation “unless a clear and compelling rationale for an alternate approach is present.” In contrast, kyphoplasty was described as a treatment option with a limited strength of recommendation. Other professional specialty organizations such as the North American Spine Society, the Society of Interventional Radiologists, and other groups did not issue recommendations against vertebroplasty or vertebral augmentation.
There are also practice patterns to consider, as the authors note. Interventional radiologists invented vertebroplasty, while orthopedic surgeons invented kyphoplasty, with each specialty retaining their dominance in the practice of the procedures they each invented. Indeed, for a time, for a time kyphoplasty training was restricted to surgeons. History, turf, and politics all too frequently intrude to prevent the rapid widespread adoption of guidelines, particularly for procedures that just seem so damned plausible. However, as I said in my article in Trends in Molecular Medicine last month and mentioned in my recent article in Nature Reviews Cancer on integrative oncology, although a prior plausibility that is clearly close to zero (as for homeopathy, for example) is a good predictor that a treatment will not be shown to work in RCTs, plausibility by itself is not anything close to a guaranteed that a procedure will be shown to work in RCTs.
Also, physicians practicing science-based medicine are human beings as well. They’re just as prone to biases and not wanting to admit that something they believed in before isn’t helping patients. This leads to a phenomenon described by the authors, but not explicitly addressed:
Although publication of the trials and the AAOS guideline influenced the use of vertebroplasty and kyphoplasty, our results indicate that both procedures are still widely used to treat patients with osteoporotic spinal fractures. Both trials were subject to criticism over their design and execution, and the appropriateness of vertebral augmentation continues to be widely debated. Multiple studies, analyses, and systematic reviews have been published since 2009 without consensus. In light of the controversy of the trials’ design and conflicting evidence from other studies, many physicians may feel free to ignore the “inconveniently negative” results from the trials and continue to offer VAPs to patients with osteoporotic spinal fractures.
You’ll note a similarity here. Remember what happened in response to the report of the Canadian National Breast Screening Study (CNBSS), which failed to find a benefit in terms of breast cancer survival due to mammographic screening. I put it all into context as best I could, but when I did that I caught some of the nastiness that was directed at the CNBSS itself and its investigators. And, of course, the VAPS studies encountered considerable criticism, nearly all of it not well-grounded in science and nearly all of it coming from radiologists and their societies
Like CAM practitioners, some have even argued that the placebo response is a legitimate endpoint for these procedures. If you don’t believe me, check out this article in Radiology by authors of one of the VAPS trials in the NEJM entitled “Vertebroplasty and the placebo response“, where such arguments are actually described. The authors conclude:
In conclusion, our interpretation of the evidence for vertebroplasty has distinctive implications for clinical practice and health policy. If the benefits of vertebroplasty derive from the placebo response, it is difficult to justify the continuing use of this procedure in clinical practice. The reason for this is not that benefit from the placebo response is lacking in therapeutic value; rather, it is doubtful that the placebo response benefits from vertebroplasty are sufficiently large to justify the risks. Moreover, although it is unknown what the outcomes would be for patients who are candidates for vertebroplasty and informed about the evidence that this procedure is no different from a sham intervention without injection of cement, they are likely to be lower than the benefits observed in clinical practice or in the Vertos II trial. Hence, on the basis of the placebo response hypothesis, the risk-benefit ratio of vertebroplasty does not appear favorable. Finally, whether continued evaluation of vertebroplasty within clinical trials might be justified is debatable—indeed, we are not in consensus on this issue.
What is being described here is the justification of a procedure that has no specific benefits but has a placebo response, which the authors refute. How many times have we here at SBM refuted the exact same sorts of arguments? It just goes to show that even ostensibly evidence-based physicians can fall prey to such arguments. Indeed, even the authors of one of the VAPS studies have implicitly accepted such arguments. The reason they reject them is not because they are not clinically significant but rather because they deem the risks of vertebroplasty not to be worth the placebo benefit.
I’ve been following the vertebroplasty story intermittently since 2005. It struck me an example of just how easily we in “conventional” medicine can fall prey to the same sort of fallacious thinking that drives the advocacy of modalities such as homeopathy or acupuncture. It’s an example that helped me in my journey towards skepticism and science-based medicine. We all like to assume that negative well-designed RCTs will result in physicians abandoning a procedure, and in general they do. However, the lesson of VAPS tells us that the process is…complicated. It’s not straightforward, and it’s messy. (There are still radiologists out there advertising vertebroplasty as if it were a miracle cure.) Still, five years on, the process for VAPS procedures appears to be well under way. Just try comparing that to the continued use of acupuncture, where no amount of negative evidence causes its abandonment, even after a period of time much, much longer than five years.

Burden of Proof Studies and Health Risk
A new way of presenting knowledge about health risks highlights the ambiguity of the evidence and the importance of basic health behaviors.
Steven Novella on October 12, 2022
Link

Much public health data comes in the form of “X increases your risk for developing Y”. Sometimes this is colloquially stated as “X causes Y”, but this is less accurate. For example, everyone by now should know that “smoking causes lung cancer” by which we mean that smoking increases the statistical risk of developing lung cancer. Most risk data is correlational, because we cannot ethically expose people to a potential risk to see how much harm it does. So really what we mean is that smoking positively correlates with the risk of developing lung cancer.
Health care providers, regulators, and the public are awash in such correlational data, with recommendations that can often be overwhelming. I previously wrote about a study in which 50 common ingredients were chosen at random out of cookbooks, and then the literature reviewed to find correlations (positive or negative) with cancer. They found that 80% of the ingredients were shown to either correlate with increased or decreased cancer risk. Often the same ingredient will have both health benefits and health risks.
Most of this is likely just statistical noise, too-eagerly reported by the media looking for sensational headlines (these common foods can kill you!!). In fact, in 2015, scientist and journalist John Bohannon deliberated carried out a crap study apparently showing that eating dark chocolate helps people lose weight. The media gullibly reported the results without the slightest investigation into the quality of the study.
There is a risk to reporting background scientific noise as if they are reliable findings that should guide our eating and health decisions. It creates a type of fatigue that can lead people to ignore actually good advice, or people who want to make an effort to be more healthy to waste time and energy on false leads. The system can be improved at every level – with better research design, clearly marking preliminary studies as such, more carefully crafted press releases (or more selective press releases), and better science journalism. But we can also use expert reviews that clear out the noise and focus attention on those risk factor correlations that deserve our attention.
The Institute for Health Metrics and Evaluation (IHME), a global health research organization at the University of Washington School of Medicine, has recently done that with a series of systematic reviews they call “Burden of Proof” reviews. They published their results in Nature Medicine, but also present their data in a useful interactive way on their website. You can, for example, look through the relative risk curves, matching possible risk factors with possible outcomes.
There are three big factors to consider when looking at any individual potential risk factor – what is the strength of the evidence, what is the magnitude of the risk, and what is the magnitude of the behavior necessary to trigger a risk. We can likely comfortably ignore alleged risks when the quality of the evidence or strength of the association is low, the magnitude of the risk is low, or the behavior would have to be extreme in order to trigger a meaningful risk.
Of course, there have been plenty of reviews looking at the evidence for apparent benefits or risks, but this effort conveniently puts a lot of information in one place in a user-friendly format with a more intuitive context.
With regard to the strength of the evidence, they use a five-star system with five stars being solid evidence, going down to strong, moderate, weak, and then possibly no association at one star. Based on this rating system alone, I think we should take four and five star associations seriously, comfortably ignore ones and twos, with three star associations being ambiguous (requires further study).
In their first crop of data they look at four risk factors: smoking, high blood pressure, eating too much red meat, and eating too few vegetables. As you can see, most of the four and five star associations have to do with smoking. One deals with high blood pressure and ischemic heart disease. There is one diet-related three-star rating, for too few vegetable and ischemic stroke, the rest of the diet associations have 1 or 2 stars. This should not be a surprise to regular readers here, or anyone very familiar with the evidence. There is very good evidence that smoking is bad for you, and that chronic high blood pressure is also a serious risk factor for vascular disease.
Diet-related risk factors, however, generally have very poor evidence. I discussed this previously with a 2019 systematic review that essentially concluded the same thing – nutritional studies tend to be of poor overall quality, mainly because they are observational with lots of noise and confounding factors.
In addition to the quality of the evidence being low, you can look at the risk curves here. If you look at smoking and cancer, there is a strong dose-response with relative risk increasing to 3-10 times baseline risk. If you look at eating red meat, the relative risks are in the 1.1-1.3 range. The “dose” also needs to be relatively high.
Looking through this data (and I am looking forward to seeing more data they plan to add) reinforces the bottom-line advice I have given in the past. Don’t go down the rabbit hole of chasing every small reported risk. Most of it is noise. What we actually know can be boiled down to some simple rules: Don’t smoke, don’t drink to excess, get sufficient sleep, exercise regularly, don’t eat too much, and eat your vegetables. When it comes to lifestyle factors, everything else is mostly noise.

The Pollyanna Phenomenon and Non-Inferiority: How Our Experience (and Research) Can Lead to Poor Treatment Choices
Clay Jones on February 14, 2014
Link
Pollyanna, a popular children’s book written in 1913 by Eleanor H. Porter, introduced the world to one of the most optimistic fictional characters ever created. She always saw the good in people and her approach to life frequently involved playing “The Glad Game”, where she attempted to find something to appreciate in every situation no matter how unfortunate. She was glad about receiving crutches rather than a doll one Christmas because it was great that she didn’t actually need them. She teaches this philosophy to those around her, even her cantankerous Aunt Polly, and the entire town is transformed into a veritable Mayberry, USA. Later, when she actually does require the use of crutches, her resolve is tested but she triumphantly finds a silver lining.
The Pollyanna principle, first described by Matlin and Stang in 1978 and also known as positivity bias, is a psychological tendency for people to place greater importance on, and assume better accuracy of, descriptive statements about them that are positive. This goes on behind the scenes while our conscious brain tends to dwell on what is perceived as negative stimuli. Though many folks do come across as pessimistic, we are subconsciously biased to accept praise and reject criticism. Anyone who isn’t clinically depressed is on some level more like Pollyanna than Eeyore.
This positivity bias also plays a large role in how we remember past events. As has been covered extensively in prior posts here on SBM, and on Dr. Novella’s excellent Neurologica blog, memory isn’t a replayed video or audio recording of prior events and our interpretations of them, but rather is a reconstruction that is prone to errors during processing and editing that accumulate over time. This leads to false memories that feel no less real than our recollection of what happened five minutes ago.
In this case, the Pollyanna principle results in positive information being more accurately processed and recalled than negative experiences. It also causes our memory of negative events to gradually become less negative as the years go by. I couldn’t have done that terribly during my first high school trumpet solo because I remember people telling me it was pretty good afterwards, right?
So what does this have to do with the practice of medicine? Biases that affect memory also impact how physicians and patients interact. I once assumed the overnight care of a child who had undergone a lumbar puncture performed by one of my female colleagues earlier that day. I ordered no tests and performed no procedures during my brief exposure to the family—yet over a year later when I admitted the same child for a completely different reason I was accused of being the terrible doctor who had unnecessarily subjected their baby to a spinal tap during the last hospitalization. Even after I showed them the documentation which proved that I had nothing to do with that (very appropriate) decision, and that I did not put a needle in their child’s spine, they refused to accept the evidence and had great difficulty trusting my diagnosis and recommendations.
There is a phenomenon in medicine, again named after the overly optimistic heroine Pollyanna, that augments the negative impact of the fallibility of human memory on medical practice with the bias towards an overly optimistic interpretation of some types of medical research and of our own clinical experience. This Pollyanna phenomenon can lead physicians to make poor choices, especially regarding which antibiotic is the best choice for a particular infection. And this is particularly common in pediatric practice, in my opinion.
The concept of a Pollyanna phenomenon in medicine has primarily been used when critiquing the approach of using increasingly broad antibiotics for the treatment of pediatric otitis media—ear infection—but it applies to a number of other bacterial infections frequently diagnosed in children. These include community acquired pneumonia, strep throat and sinusitis, three common reasons children are prescribed an antibiotic. But ear infections stand out as a clear front runner in this regard, so that is what I’ll focus on.
Ear infections are extremely common in children, again being by far the most common reason that a young child will receive an antibiotic. And even though the total number of visits for ear infections, and subsequently the total number of prescriptions for antibiotics to treat these infections, has decreased significantly over the past 20 years, still more than 80% of children will be diagnosed with one by their fifth birthday. The reasons for the decrease are myriad and include insurance issues that keep kids out of clinics, the success of the pneumococcal (PCV7 and now PCV13) and influenza vaccines, and public education campaigns aimed at teaching parents about viral infections. It’s also possible, though I have my doubts, that doctors have helped a bit by better-acknowledging other reasons for there to be fluid behind the ear drum than an acute bacterial infection.
But it isn’t all good news. According to a 2010 study in Pediatrics, there has been essentially no change at all in the percentage of visits billed for otitis media that end in an antibiotic prescription. Today, 76% percent of children diagnosed with an ear infection get an antibiotic compared to 80% in the mid 90’s. This despite the significant clinical experience of most other developed countries, reams of research papers and the 2004 publication of joint AAP/AAFP clinical practice guidelines which strongly recommended that we simply observe children over 6 months who have uncomplicated ear infections. There should have been a huge drop in the percentage of kids with ear infections receiving antibiotics but we’ve pretty much just ignored the recommendations.
The 2004 recommendations also included helpful guidelines on how to actually diagnose ear infections, particularly focusing on criteria stating that there should be more than just fluid behind the ear drum. They also required evidence of inflammation, such as redness and pain, and symptoms that are acute in onset. But there was too much wiggle room in the guidelines. They could be interpreted in such a way that children with another common condition, serous otitis media, could be diagnosed with an ear infection when presenting with fluid behind the ear drum and complaints of pain but no evidence of inflammation.
But even with this wiggle room, at the very least the days of the “asymptomatic ear infection”, an entity which both of my kids were diagnosed with at well child check-ups during infancy (both after 2004), but for which I refused treatment, should have been numbered. A new set of guidelines was published in 2013 which tightened up the criteria for diagnosis, requiring direct visualization of significant bulging of the ear drum from inflammation and purulent fluid. It will be interesting to see if we respond with a change in practice.
The 2004 guidelines also spelled out clearly what the most appropriate antibiotics for use in treating ear infections were. They further covered in what order antibiotics should be prescribed should an ear infection prove refractory to amoxicillin, the first line choice for the vast majority of cases. This was an effort to avoid overuse of unnecessarily broad agents and the risk of antibiotic resistance in the community.
Still more bad news, again coming from the widely publicized 2010 study in Pediatrics. Not only does it appear that we continued to prescribe antibiotics for any kid within arm’s reach that had fluid behind their ears, we’ve become increasingly careless in our antibiotic choices with very broad oral antibiotic agents, namely the third generation cephalosporin cefdinir (Omnicef) becoming increasingly popular. Use of cefdinir increased from 7% to 14% of all prescriptions for otitis media, frequently being used in place of amoxicillin and amoxicillin-clavulanic acid, the antibiotic recommended for severe infections or as second line for an amoxicillin treatment failure. Two silver linings did come out of the data however. Use of the recommended first line agent, the older, cheaper, more narrow spectrum and actually more effective drug amoxicillin did increase, and there was a 71% increase in the appropriate use of pain medications.
So why would a physician choose to prescribe an antibiotic, in this case a more expensive drug that has less efficacy killing the bacteria you want dead and actually increases the risk of future antimicrobial resistance in other types of infections? Well, there are a lot of reasons. Cefdinir may be more expensive, but it tastes better, can be dosed less frequently, and carries less risk of causing antibiotic associated diarrhea than amoxicillin-clavulanic acid. But in the case of ear infections, the downside of using it or a similar oral antibiotic in that class named cefixime (Suprax) outweighs the benefits.
There are more reasons, some decidedly more cynical. Drug company marketing, pressure from reps and the ease of use when there are readily available samples in the office all likely share some of the burden of blame. Still the most likely reason, if I had to hazard a guess, is more along the lines of something Pollyanna would come up with. Many physicians simply think that these antibiotics are as, or even more, effective than amoxicillin, and they really just want their patient to feel better. They want their patients’ caregivers to get sleep and be able to return to work. And they want return business too. Well maybe Pollyanna wouldn’t have come up with that last one.
But believing that an antibiotic is a better choice when it clearly isn’t is not an acceptable excuse. Are the doctors, nurse practitioners and physician assistants who write for cefdinir all incompetent? I have no doubt that some are but maybe there is reason to give them the benefit of the doubt. The deck is stacked against us a bit when it comes to these decisions after all. As I’ll get to, sometimes our own literature gives a false impression of efficacy.
Often there is a false perception that because an antibiotic is broader in its spectrum, meaning that it treats a wider variety of pathogenic bacteria, that it is better or “more powerful” than older and more narrow spectrum antibiotics. But as Mark Crislip once wrote,
the only thing in medicine that has 100% sensitivity and specificity is when a health care provider uses the terms “big-gun”, “strong” or “powerful” in relationship to antibiotics they know nothing, zip, zero, zilch about the treatment of infectious diseases.
Now I perhaps wouldn’t go so far as to say that they have no understanding of treating infectious disease. At least they are prescribing an antibiotic and not, say, a homeopathic remedy. I’ve just put The Glad Game to good use and I must say that I do feel a little less grouchy. But Dr. Crislip is absolutely right in that it’s not a good sign.
An antibiotic that is best at killing the bacteria causing the infection, and I mean really gets in there and makes them wish that their mommy-daddy had never undergone binary fission, may be extremely narrow spectrum. A staph-specific penicillin such as oxacillin is a much better choice for a sensitive staph infection, even a life threatening one, than vancomycin. It just doesn’t matter if a drug can kill 50 other types of bacteria if it can’t kill the one eroding into your mastoid process.
But sometimes it isn’t so clear that one antibiotic is better or worse than another. Both our clinical experience and the results of our research can be misleading. And so, finally, we’ve come full circle and are back to the Pollyanna phenomenon.
In 1992, a paper was published in The Journal of Pediatrics that looked at “the comparative efficacy of antibacterial agents for acute otitis media.” The authors looked at randomized, double-blinded trials of antibiotic therapy for ear infections that used both clinical and bacteriologic outcomes. Clinical success was determined by the patient’s signs and symptoms of illness resolving and bacteriologic success by the use of tympanocentesis. This involves using a needle to retrieve, in a sterile fashion, fluid from behind the ear drum that can be sent to a lab for culture of pathogenic bacteria if present. In many of the studies, tympanocentesis was performed both before and during treatment, which allowed for measuring the rates of sterilization of the fluid.
What they found is the Pollyanna phenomenon, where even the worst choices don’t seem that bad after all. For antibiotics that cleared bacteria out of the inner ear 100% of the time, the patient’s symptoms resolved 93% of the time when it was known to be caused by a bacterial infection based on tympanocentesis and 89% of the time when the ear infection was diagnosed only clinically. So roughly 1 out of 10 kids being treated with a drug that wipes the floor with the potentially offending bacterial pathogen won’t get better as expected in the real world, where tympanocentesis is rarely performed outside of an ENT office. And when a drug was used that had virtually no bacteriologic efficacy, the patient’s symptoms resolved in 71% of proven bacterial ear infections and in 74% of those diagnosed clinically.
In a nutshell, when we measure how well an antibiotic works for otitis media based on what is essentially subjective clinical improvement, drugs that kill the bacteria seem to be less effective than they really are while drugs that are one step above placebo appear to be work pretty well. This is a pattern that will always be seen when the condition being treated is largely self-limited, as ear infections have long been known to be, and may be actually caused by a viral infection which won’t respond to any antibiotic. This is why so many other countries stopped treating most ear infections with antibiotics long before our recent attempts to change, and it is why chiropractors think that their spinal adjustments work.
So it should be easy to see how an individual physician might incorrectly believe that the antibiotic they have chosen has worked, and why they might continue to use it time and time again despite it being a poor means of actually doing what they intend it to do. But there is more to the Pollyanna phenomenon. It also amounts to what is essentially a methodological “Glad Game” when designing studies that test the efficacy of a new antibiotic.
When attempting to see if a new antibiotic works, it can be challenging to design a trial that fits the mold of the randomized controlled trial that we all know and love. Ethically there can’t always be a placebo group that would leave subjects with a diagnosed bacterial infection untreated. In pediatric research, historically most antibiotic studies have not had a placebo control group and were designed as non-inferiority trials. This means exactly what it sounds like. Their goal is to demonstrate that the new drug is not unacceptably worse than the standard of care treatment, which may have also been based on non-inferiority trials. But non-inferior does not necessarily mean equivalent.
So in the case of cefdinir (#1, #2), comparisons to established treatments for ear infections have largely shown it to be comparable in effectiveness. But, as you can likely imagine, its efficacy is actually unclear because of the spontaneously-resolving nature of otitis media. When pharmacokinetics and pharmacodynamics are also looked at, true effectiveness can be triangulated.
In general, oral cephalosporins like cefdinir are not absorbed well, are easily rendered inactive by sticking to proteins in the blood, and don’t last long in the body. This leads to the level of drug available to kill the offending bacteria dropping below what is needed faster than other choices, which is a set up for resistance issues. For this and several other reasons, such as poor postantibiotic effect and tissue accumulation, cefdinir and cefixime are never better than amoxicillin or amoxicillin-clavulanic acid for killing susceptible bacteria. And the bacteria that cause the overwhelming majority of ear infections (and pneumonia) almost always are.
Non-inferiority trials are not all bad. They are necessary in many types of infections, particularly when they are not self-limited or frequently blamed when the actual culprit is a virus. It would be horribly unethical to compare the use of a new antibiotic for MRSA meningitis to placebo, and naturally a study such as that would never be approved. And if two antibiotics truly were determined to be equivalent in effectiveness, factors such as palatability and ease of dosing can be very meaningful as they may improve adherence to treatment recommendations.
But in the case of infections such as otitis media, community acquired pneumonia in young children, and acute sinusitis, the high rate of spontaneous resolution raises legitimate questions about the value of non-inferiority trial findings. Of note, in 2010 the FDA actually published nonbinding recommendations addressing this very subject. They called for increased use of superiority designs in these conditions and for better justification of the non-inferiority margins and efficacy endpoints used in trials involving infections that are not self-limited.
Conclusion
Pollyanna saw the best in every person, and found something positive in every situation. This is an admirable outlook on life, but one that doesn’t necessarily always translate well to the practice of medicine, especially when it comes to choosing the right antibiotic. We are all susceptible to the same biases and errors in logic, and not surprisingly there are similar issues in medical practice and research.
The Pollyanna phenomenon, an artifact of how we assess the success of our recommendations and study antibiotic effectiveness in largely self-limited bacterial infections, has likely led many physicians to make poor treatment choices. There is no better example of this than in the treatment of pediatric acute otitis media, where we have exposed many children to antibiotics that may only seem to be working but definitely increase the risk of developing resistance in the individual and the community. An understanding of pitfalls such as this, and awareness of the science behind expert treatment recommendations, can improve the care we provide our patients.

False “balance” on influenza with an appeal to nature
Scott Gavura on February 13, 2014
Link

One of the encouraging shifts I’ve seen in health journalism over the past few years is the growing recognition that antivaccine sentiment is antiscientific at its core, and doesn’t justify false “balance” in the media. There’s no reason to give credibility to the antivaccine argument when their positions are built on a selection of discredited and debunked tropes. This move away from false balance and towards a more accurate reflection of the evidence seems to have started with the decline and disgrace of Andrew Wakefield and his MMR fraud. And there is now no question that antivaccine sentiment has a body count: Simply look at the resurgence of preventable communicable disease. Today, antivaccinationists are increasingly recognized for what they are – threats to public health. It seems less common today (versus just 5 years ago) that strident antivaccine voices are given either air time or credibility in the media.
But false balance on topics like influenza can occur without giving a voice to groups like antivaccinationists. A more subtle technique to shift perceptions is both widespread and hard to detect, unless you’re aware of it: the naturalistic fallacy, known more accurately as the appeal to nature. In short, it means “It’s natural so it’s good” with the converse being “unnatural is bad.” In general, the term “natural” has a positive perception, so calling a product (or a health intervention) “natural” is implying goodness. The appeal to nature is so common that you may not even recognize it as a logical fallacy. Unnatural can be good, and natural can be bad: Eyeglasses are unnatural. And cyanide is natural. Natural doesn’t mean safe or effective. But the appeal to nature is powerful, and it’s even persuasive to governments. If we believe that health interventions and treatments should be evaluated on their merits, rather than whether or not they’re “natural”, then decisions to regulate “natural” products differently than the “unnatural” ones (like drugs) makes little sense. Yet the Dietary Supplement Health and Education Act was a legislative appeal to nature, introducing a different regulatory and safety standard for a group of products while drawing a fallacious distinction with “unnatural” products like drugs. Canada fell for the appeal to nature too: It has the Natural Health Products Regulations which entrenched a lowered bar for efficacy and safety for anything a manufacturer can demonstrate is somehow “natural”.
The combination of false balance and the appeal to nature has a subtle but insidious effect on how we view health care. Promoters of CAM (complementary and alternative medicine) know and use this strategy very effectively. If we’re willing to mentally distinguish between “conventional” and “natural” health approaches, then we may be more willing to overlook issues with respect to evidence – the Achilles’ heel of CAM. It’s a simple strategy for advocates to wrap their products with the “natural” banner, and our own perceptions give it a mental pass. As consumers, we’re the ones with the fallacious reasoning. The promoter is just priming us to accept the fallacy.
While I take some comfort in the fact that I’m told this blog is both widely read and well-respected, its reach is still miniscule compared to those that promote CAM. (The SBM logo should be Sisyphus, because that’s how it can feel when you’re providing science-based information against a torrent of pseudoscience.) I don’t envy Dr. Oz, but I do envy his reach and his platform, and only wish he used his prominence to promote credible health strategies, instead of what is often rank quackery. As a pharmacist I can literally see the effectiveness of the mainstream media on shaping perceptions of heath, and driving consumers to act on those demands. When I used to work in a retail pharmacy, it was immediately obvious which new miracle supplement had been hyped on Oprah, as demand would explode following an episode.
No matter how effective we are as promoters of science-based medicine, we’ll be fighting a losing battle against pseudoscience and CAM unless we have health journalists as partners. The very best health journalists are not just superb communicators; they share SBM’s goals of advocating for the best health care. My hypothesis is that the decline of major media organizations (particularly print) over the past decade may be making the task of public health and science communication more difficult. Not only do the number of dedicated health and science journalists seem to be declining, but those journalists that are tasked with covering health issues may lack the experience, background and context to communicate in the most responsible and effective way. Today it’s a hunt for eyeballs and clicks.
A series of columns in The Globe and Mail summarizes the continued challenge to communicating science-based medicine. If you’re not familiar with The Globe, it’s Canada’s largest national newspaper, and while it’s no New York Times, it’s generally considered Canada’s paper of record. The Globe has some excellent health journalists like André Picard and Carly Weeks who consistently provide some of the best health content in the country. But recently The Globe launched a new online feature, the Health Advisor, which appears to be a series of contributed posts from writers that are external to the paper:
Welcome to Health Advisor, where contributors share their knowledge in fields ranging from fitness to psychology, pediatrics to aging. Follow us @Globe_Health.
Most of the columns I’ve seen seemed to have reasonable content, and some have medically qualified authors. The series seems to be an effort to gain web traffic, as there’s no coherent theme other than the intent for posts to be clickbait, with titles such as “Former Olympian Jennifer Heil’s five tips for eating healthy on the road” or “How to tell when your memory loss is normal – and when it’s dementia.” But a recent column made me look more closely at the series: “Eight easy, natural ways to keep the flu at bay (without leaving home).” It’s written by Bryce Wylde, a homeopath who is a regular on shows like Dr. Oz. It would be a straightforward matter to look at Wylde’s recommendations and the science evidence supporting them. But the story is far more interesting than that. It starts with what Wylde suggests. He leads off with:
There are some highly effective natural solutions you can try to help prevent contracting the flu, or mitigate the symptoms once you have it. Still, recent news about the effectiveness of some supplements and home remedies has left many confused with what remedies are best to take both proactively and reactively.
Cue the appeal to nature and the false balance. His most conventional recommendation is getting more sleep. No argument there. But he goes on to suggest the following:
    • Apply petroleum jelly to the insides of your nostrils: How this would block influenza isn’t clear. Given there’s a published case report of lipid pneumonia from nasal application of petroleum jelly, and no evidence of efficacy, this seems like an irresponsible recommendation.
    • Thinly cut and plate two onions. Sprinkle sugar over the sliced onions. Cover with plastic wrap and refrigerate for two hours. With the liquid that results, dose out one tablespoon every three to four hours. Again, zero information to suggest that sliced onions have any effectiveness against influenza.
    • Like onions, honey-based homeopathics may provide a safe and effective solution. Honey may have some effectiveness to treat cough, but it does nothing to affect the risk of infection or the severity of an infection. Honey-based homeopathics are simply honey, as the homeopathic nostrums in these products are ineffective and then diluted to the point that any of the original substance is gone.
    • Garlic: What’s good for the heart blows infections apart. Yet another folk remedy with no evidence to suggest it has any effect on influenza or its complications. A systematic review of garlic for colds (not the flu) noted that claims of effectiveness are based on poor-quality data.
    • Get proactive with probiotics. Wylde gives the standard alt-med trope that “80% of your immunity is in your gut” which always makes me wonder how immunity is being quantified. He touts Bio-K brand of probiotics. Indeed, the Bio-K website claims it is helpful for influenza and points to this study as evidence. However that study used a different formulation of probiotics and looked at symptoms only, finding a modest improvement. There’s no information to demonstrate it has any effect in influenza, as the study didn’t determine the cause of each illness.
    • Shower your nose. Wylde recommends a neti-pot, which may help with the symptoms of a cold but is would have no effect on preventing or treating an influenza infection.
    • Whip up an immune-boosting soup. Yet another trope, the scientifically ignorant idea that immunity is a singular thing that can be “boosted”. Wylde recommends astragalus root with ginger and garlic. A good soup makes anyone feel better, but there’s nothing to astragalus or the other ingredients for the prevention of influenza.
By any responsible health journalism standard, this has serious problems. I was not alone in noticing Wylde didn’t actually recommend strategies that do have some evidence to support them, such as vaccination, handwashing and social isolation. When I put the question to The Globe on Twitter, Wylde responded and claimed that the mention of vaccines had been removed at The Globe’s request, which I found pretty concerning. And then The Globe responded:
And then it doubled down on the column, promoting it all day:
Spot the fallacies? A lot of people did, and commented online and via Twitter. The Globe’s response was more problematic than Wylde’s – it ought to know better. To The Globe’s credit, they subsequently took the feedback seriously enough for the paper’s “public editor” to address it a few days later, in a column entitled Homeopath’s advice needs to be balanced:
Last Friday, a blog on natural ways to keep the flu at bay written by an alternative-medicine specialist garnered criticism on social media and in the story comments. The blog suggested a variety of solutions that the writer, who has a diploma in homeopathic medicine, felt could either help to prevent contracting the flu or mitigate the symptoms. The blog was criticized by medical professionals on a number of points and most said it missed the key advice that what really works is the flu vaccine.
One public health physician-in-training wrote to me to say the article “makes a number of health claims regarding ways to prevent or treat the flu, but which lack any evidence to support them. Indeed, [the] … recommendations (such as the use of probiotics) …demonstrate a basic misunderstanding of what the flu is (a virus, not a bacteria). “As a member of the medical community, it concerns me when unfounded claims from Complementary & Alternative Medicine (CAM) providers are presented as fact and given legitimacy via publication. Incomplete or incorrect health information can produce very real harm, as it often prevents people from seeking out appropriate, evidence-based care (e.g. flu vaccine, antiviral medications).”
The physician’s question was about The Globe and Mail’s obligation in presenting health information. It is an excellent question. Medicine is a science based on studies and factual observations, unlike, for example, politics, where it is important for media coverage to balance the different sides of a debate.
The Globe has published many articles on the importance of getting a flu vaccination. But if the paper chooses to run an article by a homeopath in addition to the articles by medical professionals, such as a doctor, nurse or pharmacist, the article needs to be seen in the context of who the writer is and what his credentials are.
The article properly sets Health Advisor in context, saying this is a blog where contributors share their knowledge. At the bottom in italics is an explanation of the credentials of the writer so that readers will see this person is not a medical doctor but rather an alternative-medicine expert.
However, in my view, the label Special to The Globe and Mail is not clear, as it is commonly used to indicate articles by freelance journalists. It also would have been preferable to have the related links to the story balance the homeopath’s advice with a medical professional’s advice. The top three links were about learning to dream, outdoor fun and good germs. In my view, the top link should have been this article: “As a doctor, I’ve seen why vaccination is a ‘no brainer.’ ”
I think there should be a bias toward medical professionals writing about medicine and, while there is room for some coverage on alternative or homeopathic treatments, care should be taken to always balance such coverage with a doctor’s or other medical professional’s experience and expertise.
Regrettably, this response seemed to miss the point of the criticism, suggesting that all the Globe needs to do is to “balance” columns like this with a bit more science. I see the issue of “freelance journalist” as a bit of a red herring – this is the Globe and Mail, and the paper shouldn’t be lending its credibility to contributors touting unscientific health information. The proper way to deal with pseudoscience isn’t to “balance” it with links to credible health information, it’s to present the facts accurately the first time, and not give a platform to quackery. And when it comes to CAM, the science is very clear: if a “natural” approach or remedy has medicinal effects, it’s simply medicine. (All the rest is a nice bowl of soup and some potpourri, argues Dara O’Briain). There is no need to present two different perspectives. Just describe what’s effective.
Conclusion
The appeal to nature is one of the most insidious logical fallacies, and it can contribute to misguided attempts to “balance” science-based information against treatments and strategies that lack good evidence of effect. While The Globe and Mail is to be commended for publicly acknowledging and responding to online criticism, a stronger response would have acknowledged the paper’s responsibility to ensure that contributed health content is reviewed and vetted by the appropriate subject-matter experts. Logical fallacies are critical thinking “blind spots”, and it’s only when we know they exist that we can spot them.

On the “right” to challenge a medical or scientific consensus
While everyone has the “right” to challenge a scientific consensus, overthrowing a scientific consensus takes data. Lots and lots of data. Few people realize how difficult it is.
David Gorski on March 23, 2015
Link

The major theme of the Science-Based Medicine blog is that the application of good science to medicine is the best way to maintain and improve the quality of patient care. Consequently, we spend considerable time dissecting medical treatments based on pseudoscience, bad science, and no science, and trying to prevent their contaminating existing medicine with unscientific claims and treatments. Often these claims and treatments are represented as “challenging” the scientific consensus and end up being presented in the media—or, sadly, sometimes even in the scientific literature—as valid alternatives to existing medicine. Think homeopathy. Think antivaccine views. Think various alternative cancer treatments. When such pseudoscientific medicine is criticized, frequently the reaction from its proponents is to attack “consensus science.” Indeed, I’ve argued that one red flag identifying a crank or a quack is a hostility towards the very concept of a scientific consensus.
Indeed, I even cited as an example of this attitude a Tweet by Jane Orient, MD, executive director of the American Association of Physicians and Surgeons (AAPS). This is an organization of physicians that values “mavericky-ness” above all else, in the process rejecting the scientific consensus that vaccines are safe and effective and do not cause autism or sudden infant death syndrome (SIDS), that HIV causes AIDS, and that abortion doesn’t cause breast cancer, to name a few. Along the way the AAPS embraces some seriously wacky far right wing viewpoints such as that Medicare is unconstitutional and that doctors should not be bound by evidence-based practice guidelines because they are an affront to the primacy of the doctor-patient relationship and—or so it seems to me—the “freedom” of a doctor to do pretty much damned well anything he pleases to treat a patient.
I’ll repost Dr. Orient’s Tweet:
@secularbloke @gorskon @AAPSonline Skepticism is the essence of science; consensus is its death
— Jane Orient, MD (@jorient) December 1, 2014
As I said at the time (a great example can be found here), on the surface this seems quite reasonable, but, as I’ve discussed on many occasions, science is all about coming to provisional consensuses about how the universe works. Such consensuses are challenged all the time by scientists. Sometimes they are shown to be incorrect and require revision; sometimes they are reinforced. That’s how science works.
The reason I brought up this issue again is because I came across a couple of articles relevant to this topic. The first is one by John Horgan, who blogs over at Scientific American, entitled ‘Everyone, Even Jenny McCarthy, Has the Right to Challenge “Scientific Experts”.’ Having a tendency towards snarkiness, my first thought was to simply dismiss this as a straw man argument (at least the title), because I know of no strong defender of science (least of all I) saying that non-experts—yes, even Jenny McCarthy—don’t have the right to challenge experts. When we complain about “false balance,” it’s not because we think that, for example, antivaccine activists don’t have the “right” to challenge the experts supporting the scientific consensus. Rather, it’s because we argue—correctly, I believe—that media outlets all too often present such challenges as falsely equivalent to the actual consensus science being challenged, in essence, putting someone like Jenny McCarthy on or near the same plane as actual scientists. Examples abound and have been discussed on this very blog, embracing many relevant topics, such as influenza, dubious cancer cures, homeopathy, vaccine safety and efficacy, fear mongering about food by Vani Hari (a.k.a. The Food Babe) and many other topics.
So let’s see what Horgan objects to:
Years ago I was blathering to a science-writing class at Columbia Journalism School about the complexities of covering psychiatric drugs when a student, who as I recall had a medical degree, raised his hand. He said he didn’t understand what the big deal was; I should just report “the facts” that drug researchers reported in peer-reviewed journals.
I was so flabbergasted by his naivete that I just stared at him, trying to figure out how to respond politely. I had a similar reaction when I spotted the headline of a recent essay by journalist Chris Mooney: “This Is Why You Have No Business Challenging Scientific Experts.”
Mooney is distressed, rightly so, that many people reject the scientific consensus on human-induced global-warming, the safety of vaccines, the viral cause of AIDS, the evolution of species. But Mooney’s proposed solution, which calls for non-scientists to yield to the opinion of “experts,” is far too drastic.
Oddly enough, the article by Chris Mooney cited by Horgan isn’t particularly recent. It’s close to 10 months old. Be that as it may, Yes, that hapless student did have a rather naïve attitude, but what he said is not quite the same as what Mooney argued, Horgan’s conflation of the two notwithstanding. Let’s put it this way. There’s a not-insignificant difference between saying “you have no business challenging scientific experts” and “you have no right to challenge scientific experts.” The first is a warning to lay people and people without the appropriate expertise about why they should be very careful challenging a scientific consensus without saying that they have no right to make such challenges. What Mooney calls for is the recognition that there is such a thing as expertise and challenging it requires more than just a Google education.
Mooney’s article relied heavily on the viewpoint of Professor Harry Collins of Cardiff University, who runs the Centre For The Study Of Knowledge Expertise Science at the Cardiff School of Social Sciences. Specifically, Mooney was discussing a book by Collins entitled Are We All Scientific Experts Now? In the book, Collins lays out a robust defense of scientific expertise. According to Mooney, Collins is known for his investigation in which he was embedded for over a decade within the community of gravitational wave physicists to the point where he became so familiar with their culture that he was actually able to trick physicists into thinking he was one of them. During his time he refuted several myths about science, such as the “eureka moment” and the idea that scientists always follow the data where they lead when in fact sometimes they cling to established paradigms in the face of new evidence. As Mooney put it, the upshot was that “while the scientific process works in the long run, in the shorter term it is very messy—full of foibles, errors, confusions, and personalities.”
I’ve said almost exactly the same thing myself on more occasions than I can remember. In the short term, science can be incredibly messy. Early results that seemed promising often undergo a “decline effect” and appear less solid. In medicine, in particular, physicians and scientists sometimes cling to old paradigms longer than they should in the face of new evidence. On the other hand, because human lives are at stake, this is somewhat understandable. Because being wrong about new findings in medicine can cause actual harm to human beings, physicians tend to be conservative and need a lot of convincing. Sometimes this tendency goes too far. Indeed, we have a bit of a joke in medicine that no medical treatment is ever entirely abandoned until the last group of physicians who trained when it was the standard of care has retired or died off. It’s actually not quite that bad, and doctors do pay attention to negative studies. There are lots of things we as surgeons did in the 1990s as part of breast cancer treatment, for instance, that we no longer do and things that we didn’t do then (and hadn’t even thought of yet) that we do now. There’s also a countervailing tendency among physicians to “jump on the bandwagon” of new treatments before they’re properly validated, sometimes for marketing advantage, sometimes just to be a trailblazer. Laparoscopic cholecystectomy in the early 1990s comes immediately to mind as an example of this tendency.
Still, as messy as science is, I agree with Mooney that in the long term the scientific process works. I also like the Collins’ concept of the Periodic Table of Expertises, described by Mooney:
Read all the online stuff you want, Collins argues—or even read the professional scientific literature from the perspective of an outsider or amateur. You’ll absorb a lot of information, but you’ll still never have what he terms “interactional expertise,” which is the sort of expertise developed by getting to know a community of scientists intimately, and getting a feeling for what they think.
“If you get your information only from the journals, you can’t tell whether a paper is being taken seriously by the scientific community or not,” says Collins. “You cannot get a good picture of what is going on in science from the literature,” he continues. And of course, biased and ideological internet commentaries on that literature are more dangerous still.
Interactional expertise requires deep experience with a specialty of the kind that can’t be simply learned by reading the literature; in essence, it involves, to some extent or other, actual experience studying and doing research in the relevant specialty. Our professors in medical school often pointed out that at least half of what we were being taught will be incorrect in a decade. Whether that exact estimate is true or not is not critical to the main point, which is that medicine and medical knowledge changes fairly rapidly as new scientific findings are reported and that we, as physicians, have to learn to adapt and integrate these new findings into our practices. Because it’s not uncommon for keeping up with the latest literature to be too much for one person to do without help, we rely on the society of physicians and medical scientists. One way we do this is to attend medical and scientific conferences relevant to our specialty. (For example, I will be going to Houston later this week to attend the Society for Surgical Oncology meeting, and in April I will be attending the American Association for Cancer Research annual meeting in Philadelphia.) We form professional societies who gather groups of experts together to produce and periodically update guidelines based on the best existing evidence.
To show the difference between literature knowledge and interactional knowledge, Collins uses this analogy:
The next step after popular understanding is the kind of knowledge that comes with reading primary or quasiprimary literature. We will call it ‘primary source knowledge.’ Nowadays the internet is a powerful resource for this kind of material. But even the primary sources provide only a shallow or misleading appreciation of science in deeply disputed areas though this is far from obvious: reading the primary literature is so hard, and the material can be so technical, that it gives the impression that real technical mastery is being achieved.
It may that the feelings of confidence that come with a mastery of the primary literature is a factor feeding into the ‘folk-wisdom view’ [the view that ordinary people are wise in the ways of science and technology]. But any amateur trying to apply the knowledge gained from car-repair manuals will soon learn the bitter lesson that much less can be done as result of reading information than appears to be the case. The same applies to doctoral students in the sciences; their first experience of real research is usually a shock, however well accomplished they have become in reading the published literature and doing well-rehearsed experiments as undergraduates. But even experienced scientists tend not to understand the amount of tacit knowledge on which their abilities depend. Thus, studies of tacit knowledge transmission show, inter alia, that scientists will embark confidently on an experimental project having done nothing more than read the literature and only later discover the degree of joint practice and/or linguistic socialisation that is needed to make a success of it (to generate the capacity to do the thing rather than talk about it).13 Given trainee scientists’ experiences, and professional scientists’ lack of reflective appreciation of their own tacit knowledge, it is no surprise that a member of the public encountering the professional journals or the internet might easily come to think that they have found a direct line to understanding.14
Footnote 14 brings this concept home to medicine:
14. A familiar image is today’s informed patient visiting their doctor armed with a swathe of material printed from the internet. While this kind of information gathering, especially in the context of a support or discussion group, can be valuable, it is important not to lose sight of what sociologists have shown: a great deal of training and experience is needed to evaluate such information. Sociologists of science seem to forget the lessons of their own subject rather easily.
Exactly.
None of this is to say that those who haven’t reached the level of interactional knowledge about a subject don’t have the right to criticize scientific consensuses within that subject. It does, however, warn those who would be critical to avoid the hubris of thinking that their popular science or even primary literature knowledge is sufficient.
It’s also important to remember that there are scientific consensuses and then there are scientific consensuses. What I mean is that some consensuses are stronger than others, something Horgan seems to ignore or downplay. For example, he seems quite pleased with himself when seemingly he got something right that Stephen Hawking got wrong. Last year, cosmologists overseeing a project called Background Imaging of Cosmic Extragalactic Polarization 2 (BICEP2) reported the “first direct evidence” of inflation, a theory which says that the universe went through a period of extremely rapid expansion right after the Big Bang. Hawking had made a bet with another scientist, cosmologist Neil Turok, director of the Perimeter Institute in Canada, that gravitational waves from the first fleeting moments after the Big Bang would be detected by BICEP2. He had even declared victory on the BBC, leading Horgon to nearly strain his shoulder and elbow patting himself on the back when Hawking turned out to be wrong:
No less an authority than Stephen Hawking declared that the BICEP2 results represented a “confirmation of inflation.” I nonetheless second-guessed Hawking and the BICEP2 experts, reiterating my long-standing doubts about inflation. Guess what? Hawking and the BICEP2 team turned out to be wrong.
I’m not bragging. Okay, maybe I am, a little. But my point is that I was doing what journalists are supposed to do: question claims even if–especially if—they come from authoritative sources. A journalist who doesn’t do that isn’t a journalist. He’s a public-relations flak, helping scientists peddle their products.
Here’s the thing. There’s a huge difference between a well-settled scientific consensus and cutting edge cosmology. Yes, Stephen Hawking is undeniably an expert, but he was expressing a scientific opinion on a matter that was (and is) not at all close to settled science. That he turned out to be wrong is not shameful. It turns out that the BICEP2 investigators teamed up with another group of scientists from the European Space Agency to analyze the data from BICEP2 and ESA’s Planck satellite and found that the previous analysis had overlooked factors that could produce a false positive. Again, this is the sort of reversal that is not uncommon when scientists are doing research at the bleeding edge of scientific discovery.
It is a very different thing than the science that tells us homeopathy can’t work, that vaccines are safe and effective, and that energy healing is more magic than science. Moreover, as one commenter pointed out to Horgan, the “first meaningful (i.e. based on actual scientific arguments) doubts about the BICEP2 results I saw were on the blogs of professional cosmologists, and not issues raised by journalists themselves (although yes, they were later reported by journalists).” Also, as another commenter pointed out, just because the BICEP2 team was wrong about the interpretation of their experiment doesn’t necessarily mean they were wrong about inflation. That commenter also pointed out (as I just did) that inflation is not yet consensus theory.
Horgan goes on to argue that “it’s precisely because we journalists are ‘outsiders’ that we can sometimes judge a field more objectively than insiders.” Well, yes and no. It might well be possible for an outsider like Horgan to judge conflicts of interest, disclosed or undisclosed, better than “insiders” can, as in the case of pharmaceutical funding and influence on drug development. Indeed, just check out Brian Deer’s excellent work outside his other excellent work exposing Andrew Wakefield’s scientific fraud (e.g. his work on the TGN 1412 clinical trial and ). Just look at what Ben Goldacre has accomplished. That being said, unless Horgan has reached the level of interactional knowledge, his “insights” about a scientific field, in particular the scientific consensuses within that field, should not be treated the same as those of real experts.
Unfortunately, Horgan rather misses the point in his conclusion:
Google is reportedly working on algorithms for evaluating the credibility of websites based on their factual content. But there will never be a foolproof way to determine a priori whether a given scientific consensus is correct or not. You have to do the hard work of digging into it and weighing its pros and cons. And anybody can do that, including me, Mooney and even Jenny McCarthy.
By the way, I think McCarthy grossly overstates the dangers of vaccines–I’m glad my kids got vaccinated–but I, too, have concerns about some vaccines.
Reading Horgan’s article referenced in the last sentence, “Michele Bachmann Wasn’t Totally Wrong about HPV Vaccines“, made me cringe. Although Horgan does point out that he didn’t believe Bachmann’s ignorant blather about Gardasil causing mental retardation, he does seem to conflate questions of Merck’s marketing strategy for Gardasil with whether a mass vaccination campaign to prevent HPV is worth the expense and bother, seeking to poison the well about Gardasil based on distrust of pharmaceutical marketing instead of concentrating on the scientific and medical merits and disadvantages of the vaccine.
A number of commenters had excellent retorts to Horgan’s argument that anyone, including Jenny McCarthy, Chris Mooney, and yes, John Horgan, can do the “hard work of digging into” a scientific consensus and “weighing its pros and cons”: And anybody can play basketball, including me, Michael Jordan, and Stephen Hawking.
Or, with the country being in the thick of March Madness and all:
People can pretend to be scientists. That doesn’t make them scientists. They can forget that they failed high school math, majored in art history in college, and became an actor. I can also forget that I did not play for the Tar Heels and the Bulls, but basketball sure is fun. Expertise takes a lot of hard work. Someone claiming they have expertise when they don’t is arrogant. James Inhofe melting a snowball is not expertise in climatology. Call it a gimmick, but it was a gimmick that showed he did not know the difference between global climate and local weather. Yes, he does have the right to his beliefs, but others have the right to laugh at his beliefs as being profoundly ignorant.
Precisely. This is where Horgan again misses the point. It’s not about people without expertise not having the “right” to question a scientific consensus. Clearly, a fair reading of his article indicates that not even Mooney meant that when he said you have “no business” questioning scientific consensus. (Besides, as I recently learned when I was published in Slate.com, it’s usually the editor who comes up with the headline, not the writer.) Rather, it’s about how that consensus is questioned. When it’s questioned, as Jenny McCarthy and antivaccinationists question scientific consensus, using misinformation, pseudoscience, cherry picked studies, and misinterpretation of other scientific studies, such “questioning” devolves into denialism and should be called out. In other words, how one questions a scientific finding matters. A lot.
Finally, it’s also about how much the questioning of a scientific consensus by a non-expert should be valued. Someone like Horgan might have a modicum of credibility questioning a scientific consensus based on his experience as a science journalist, particularly when we’re talking about something that isn’t a particularly strong consensus (inflation), if it’s even consensus at all. Someone like Jenny McCarthy, with no relevant expertise even reaching the level of “literature knowledge,” has no credibility at all. Having people like me say so and people like Mooney saying that she has “no business” making such pronouncements is simply the price she pays for parading her ignorance to the world, particularly when her ignorance contributes to real degradations in public health through increasing numbers of parents not vaccinating their children. In the end, what Horgan seems to be arguing is that we should take pseudoexpertise seriously. I disagree.
Related posts:
     1.  Hostility towards scientific consensus: A red flag identifying a crank or quack
     2.  Science-based medicine, skepticism, and the scientific consensus
     3.  Pseudo-expertise versus science-based medicine
     4.  The “decline effect”: Is it a real decline or just science correcting itself?

   “There must be a reason,” or how we support our own false beliefs
David Gorski
Link
For a change of pace, I want to step back from medicine for this post, although, as you will see (I hope), the study I’m going to discuss has a great deal of relevance to the topics covered regularly on this blog. One of the most frustrating aspects of being a skeptic and championing science-based medicine is just how unyielding belief in pseudscience is. Whatever realm of science in which there is pseudoscience I wander into, I find beliefs that simply will not yield to science or reason. Whether it be creationism, quackery such as homeopathy, the anti-vaccine movement, the “9/11 Truth” movement, moon hoaxers, or any of a number of pseudoscientific movements and conspiracy theories, any skeptic who ventures into discussions of such a topic with believers will become very frustrated very fast. It takes a lot of tenacity to keep going back to the well to argue the same points over and over again and refute the same nonsense you’ve refuted over and over again. Many do not have sufficient stick-to-it-iveness, leading them to throw up their hands and withdraw from the fight.
Although some of us here have blamed this phenomenon on “cultishness” and, make no mistake, I do think that there is an element of that in many of these movement, particularly the anti-vaccine movements, cultishness alone can’t explain why people hold on so hard to beliefs that are clearly not supported by science or evidence, such as the belief that vaccines are responsible for an “autism epidemic.” Then last week, what should pop up in the newsfeeds that I regularly monitor but a rather interesting article in Science Daily entitled How We Support Our False Beliefs. It was a press release about a study1 that appeared a few months ago in Sociological Inquiry, and the the study was described thusly:
In a study published in the most recent issue of the journal Sociological Inquiry, sociologists from four major research institutions focus on one of the most curious aspects of the 2004 presidential election: the strength and resilience of the belief among many Americans that Saddam Hussein was linked to the terrorist attacks of 9/11.
Although this belief influenced the 2004 election, they claim it did not result from pro-Bush propaganda, but from an urgent need by many Americans to seek justification for a war already in progress.
The findings may illuminate reasons why some people form false beliefs about the pros and cons of health-care reform or regarding President Obama’s citizenship, for example.
Let me make one thing plain right here and right now, before I delve into the study itself. This analysis has very little to do with whether invading Iraq preemptively was a good idea or not, whether it was right or not. I am not even going to address that question, as it is not an appropriate topic for SBM. However, it does use as a model the widespread belief in from 2001 to 2004 that Iraq’s ruler Saddam Hussein had aided and abetted the terrorists who on 9/11/2001 crashed jet liners into the World Trade Towers, the Pentagon, and into the fields of Pennsylvania when the passengers fought back. When I read this study, I immediately realized that its results might be more generalizable and in particular apply to the movements to promote unscientific medical practices that we routinely encounter on SBM. The widespread belief that persists even to this day that Saddam Hussein somehow had a hand in 9/11 is very much like the beliefs of the anti-vaccine movement in its resistance to evidence is a lot like belief in quackery and other pseudoscience. As one of the investigators, Steven Hoffman put it, “”This misperception that Hussein was responsible for the Twin Tower terrorist attacks was very persistent, despite all the evidence suggesting that no link existed.”
One problem with this study is that the authors sidestep the question of whether this mistaken belief was due to propaganda or not. Although it has been argued by many that the Bush administration promoted such a connection by innuendo and misinformation, this study suggests is that there was a lot more to it than that. However, in a way, that’s not a huge problem, as this study is not so much about how false beliefs come to be widespread in the first place but rather why human beings keep holding on to them after evidence showing them not to be true is presented to them. As in the case of quackery, how belief in pseudoscience comes to be and why humans continue to cling to that belief even in the case of overwhelming contradicting evidence are two separate, but related, problems. Right in the introduction, the authors of this study lay the conflict on the line in a way that one could see as relevant to SBM as well:
Explanations for this have generally suggested that the misperception of a link resulted from a campaign of innuendo carried out by the Bush administration that explicitly and implicitly linked Saddam with Al Qaeda. For example, Gershkoff and Kushner (2005:525) argue that “the Bush administration successfully convinced [a majority of the public] that a link existed between Saddam Hussein and terrorism generally, and between Saddam Hussein and Al Qaeda specifically.” We characterize this explanation as being about the information environment: it implies that if voters had possessed the correct information, they would not have believed in the link. Underlying this explanation is a psychological model of information processing that scholars have labeled “Bayesian updating,” which envisions decision makers incrementally and rationally changing their opinions in accordance with new information (Gerber and Green 1999).
In this article we present data that contest this explanation, and we develop a social psychological explanation for the belief in the link between Saddam and Al Qaeda. We argue that the primary causal agent for misperception is not the presence or absence of correct information but a respondent’s willingness to believe particular kinds of information. Our explanation draws on a psychological model of information processing that scholars have labeled motivated reasoning. This model envisions respondents as processing and responding to information defensively, accepting and seeking out confirming information, while ignoring, discrediting the source of, or arguing against the substance of contrary information (DiMaggio 1997; Kunda 1990; Lodge and Tabor 2000).
It was with some trepidation that I decided to delve into this study because (1) I’m not a sociologist and (2) I realize that this particular comparison may ruffle some feathers, particularly any readers with conservative political leanings, particularly since other examples mentioned in the article are from the current debate about health care (“death panels,” anyone?) or the “birther” movement, namely that group of people who believe that Barack Obama is not a natural born citizen of the United States and therefore is ineligible to be President, a belief that defies all evidence and reason. However, again, one can see how this sort of a model could apply very well to belief in all manner of pseudoscience, the most prominent at the moment being the anti-vaccine movement. In fact, this model applies to all of us, and the choice of subjects was more for convenience than out of bias, as the authors take pains to point out that these sorts of flaws in reasoning are widespread among all political ideologies:
Our choice of subjects should not be taken to imply that the processes we are examining here are particular to conservatives: we expect that, had we conducted this study in the late 1990s, we would have found a high degree of motivated reasoning regarding the behavior of President Clinton during the Lewinsky scandal. Previous research on motivated reasoning has found it among respondents of all classes, ages, races, genders, and affiliations (see Lodge and Tabor 2000).
But what is this “motivated reasoning” about which the authors write and how did they look at it? What exactly did the authors do to test their hypothesis? The authors base their concept on that of cognitive dissonance. Delving back into my medical school psychology courses, I remember that cognitive dissonance derives from the observation that people do not like to be aware when they hold contradictory beliefs. Indeed, it is the name given to the feeling we have when we are made aware that we are holding two contradictory thoughts at the same time, and the strength of the dissonance depends upon the importance of the subject to an individual, how sharply the dissonant thoughts conflict, and how much the conflict can be rationalized away. Cognitive dissonance theory thus posits that, when faced with evidence or occurrences that challenge their beliefs, people will tend to minimize the dissonance any way they can without giving up those beliefs.
One classic example of cognitive dissonance often cited include smokers who try to rationalize their unhealthy habit. An even more classic example, described by Leon Festinger in the book When Prophecy Fails is the existence of “end of the world” cults, who believe that the world will end on a certain date. Festinger infiltrated a cult that believed the world was going to be destroyed and its leader and followers would all be taken away by an extraterrestrial visitor before dawn on December 21, 1954. Dawn came and went with no visitor and no world-destroying cataclysm. Did the group disintegrate? That’s what you might think it would do, but instead it concluded that the cataclysm had been called off and that God stayed his hand because the cult had “spread so much light.” Thus, the strategy for eliminating the cognitive dissonance between the cult’s beliefs and the undeniable fact that the world had not ended, nor had an alien visitor come, at the predicted time was not to conclude that the cult’s beliefs had been wrong, but rather to conclude that the cult had somehow stopped the catastrophe through its righteousness.
Now, on to the study. The authors chose a study population from precincts in counties that had voted most heavily for Bush in 2000, identifying them through voter registration records. Surveys were mailed to 1,062 voters, of which 12 were returned to sender. Of the remaining 1,050, 267 responded, for an overall adjusted response rate of 25.4 percent. Of these surveys, 21 were unusable in this study, so the analysis of the surveys is based on 246 respondents. Subjects who agreed to be interviewed (84, of which 49 met the study criteria of having voted for Bush and believed that Saddam Hussein was somehow involved in the 9/11 attacks) were then subjected to what the authors termed a “challenge interview” to determine whether they were exhibiting Bayesian updating (the willingness to change one’s mind in the face of contradictory information from a trusted source) or motivated reasoning (resisting contradictory information). The exact wording of the challenge was, “..let’s talk about Iraq. As you see in these quotes, the 9/11 Commission found that Saddam Hussein was not behind the September 11 attacks. President Bush himself said, “This administration never said that the 9/11 attacks were orchestrated between Saddam and Al Qaeda.” What do you think about that? [show newspaper clips]”
Data were analyzed as follows:
First, we examined whether our respondents deflected the information, and we categorized the strategies that they used to do so. Second, to conduct a more stringent test of the motivated reasoning hypothesis, we examined whether respondents attended to the contradictory data at all. Lupia (2002) argues that Bayesian updating happens in three stages: to successfully change opinion, a piece of information must be attended to, remembered, and used in decision making. The first stage, attention, is a prerequisite for the second and third stages. By coding whether our respondents attended to the information we produced a minimum estimate for motivated reasoning, which can also happen at the second or third stages.
What did the authors find? Basically, only 2% used Bayesian updating, changing their mind about whether in response to information provided that challenged their belief. 14% denied that they actually believed in a link at all, even though they had responded in writing in the survey that they believed there had been a strong link between Saddam Hussein and 9/11. Most of the others ignored the evidence against their belief and launched straight into arguments of why the war against Iraq was justified, thereby reducing their cognitive dissonance by asserting, in essence, that the war was a good idea even if Saddam Hussein had not been involved in 9/11. While this study had a fair number of weaknesses inherent in studies of this sort, overall it’s not bad. For example, it didn’t look at whether the survey and challenge interviews changed any minds later, after the respondents left. Nor is it clear, given that the interviews were carried out in 2004, whether this study accurately showed the origin of the erroneous beliefs about Saddam Hussein having helped al Qaeda. Rather, it probably showed more of the reasons for the persistence of such beliefs, regardless of how they formed. However, as I mentioned before, this is probably not a huge flaw in that the study was more designed to look at how people support their false beliefs, rather than how they formed those beliefs in the first place.
From their interviews, the authors postulated a mechanism the refer to as inferred justification, which the authors describe thusly:
Finally, our interviews revealed an interesting and creative reasoning style that we call inferred justification: recursively inventing the causal links necessary to justify a favored politician’s action. Inferred justification operates as a backward chain of reasoning that justifies the favored opinion by assuming the causal evidence that would support it. As with the situational heuristics described above, respondents begin with the situation and then ask themselves what must be true about the world for the situation to hold.
None of this should come as any surprise to skeptics and supporters of science-based medicine. It’s nothing more than a fancy way of describing the flaws in reasoning that virtually all boosters of unscientific medicine use: post hoc ergo propter hoc reasoning and cherry picking the small amount of evidence that supports their belief and ignoring the vast majority of the evidence that does not. Forget politics. Forget “liberal” versus “conservative.” All this study is saying is that people who have deep emotional ties to a belief will try very hard not to have to give up that belief. I realize that the example used for the study may irritate some of our readers, but, even so, the results and conclusions shouldn’t come as any surprise and I hope that the example chosen will not lead some to reject the conclusions out of hand. In fact, we see the same sorts of “inferred justification” all the time in those who should know better regarding all sorts of topics. If Tom Harkin, for example, were to take a similar survey followed by a challenge interview chock full of posts from SBM that showed the “alternative medicine” that he believed in doesn’t work, I have no doubt that it would have no effect on his advocacy for NCCAM (in fact, it didn’t; when confronted with a string of negative studies from NCCAM, Harkin simply complained that NCCAM’s studies are all negative) or his trying to slip provisions into the health care reform bill that would require the government to pay for “alternative” medicine.
Be that as it may, let me give you an excellent example in a realm we deal with regularly: the antivaccine movement. Let’s get a bit more specific than that. In the U.S., in the late 1990s through the middle part of this decade, the anti-vaccine movement latched on to the mercury-containing preservative (thimerosal) used in many childhood vaccines as a cause of autism. They based this belief on the correlation between a rise in the number of autism diagnoses beginning in the early to mid-1990s and the expansion of the vaccine schedule to include more vaccines, even going so far as to call the rise in autism diagnoses an “autism epidemic.” In 1999, despite little evidence that thimerosal-containing vaccines (TCVs) were in any way associated with autism or other harm, authorities ordered the phase-out of TCVs in favor of thimerosal-free alternatives. By the end of 2001, the only childhood vaccines still containing thimerosal were flu vaccines, and few children received them. The total dose of mercury received by children due to vaccines plummeted precipitously to levels not seen since the 1980s. And what happened to autism diagnoses?
They continued to rise, which is why, very early on in my tenure here at SBM, I wrote a post entitled Mercury in vaccines as a cause of autism and autism spectrum disorders (ASDs): A failed hypothesis. It’s unequivocal that autism rates have continued to rise since 2001, and alternative explanations for the rise in autism rates are quite compelling, specifically broadening of the diagnostic criteria, diagnostic substitution, and increased awareness, along with evidence that, correcting for these changes, the “true” prevalence of autism has probably not changed much over decades; i.e., there is no “epidemic” of autism. In response, some who cling to the mercury hypothesis claim that even a trace of mercury would still cause this autism “epidemic,” failing to explain why we didn’t see such an epidemic before decades ago when thimerosal was first used in a relatively few vaccines. Other members of the anti-vaccine movement have moved on to more difficult-to-falsify hypotheses, such as different “toxins” in vaccines (the latest of which is squalene) or the concept that when it comes to vaccines we are giving “too many too soon.” The reason, of course, is because it is the belief that vaccines cause autism and all sorts of other harm that is driving the anti-vaccine movement. When more and more evidence fails to support this belief, rather than giving up the belief, the anti-vaccine movement either ignores the data and moves on to another hypothesis, expecting science to play Whac-A-Mole with each new outlandish claim; dismisses or minimizes it as being due to a conspiracy by big pharma to hide The Truth About Vaccines; or finds a way to twist the science to be either neutral or even supportive of its ideas. In the process, the anti-vaccine movement does what this study describes very well: It infers justification by recursively inventing links between vaccines and autism.
Another good example is the reaction to revelations about Andrew Wakefield. Andrew Wakefield, as you may recall, is the British gastroenterologist who in 1998 published a study in The Lancet that claimed to find a link between the MMR vaccine and “autistic enterocolitis.” This study, aided and abetted by truly irresponsible journalism, launched a panic in the U.K. that is only now starting to abate. In the interim, measles, once thought conquered, has become endemic again in the British Isles. In any case, it matters not to the anti-vaccine movement that (1) Wakefield’s Lancet study was poorly designed and utterly refuted by later studies; (2) investigative journalist Brian Deer discovered and published that Wakefield received £435,643 in fees, plus £3,910 expenses from lawyers trying to show that the MMR was unsafe; (3) the PCR laboratory that Wakefield used for his work was so poorly run that it apparently had no knowledge of the concept of a negative control; and (4) in 2009 Brian Deer unearthed and published evidence that strongly suggests that Wakefield had falsified data in his 1998 Lancet paper. Instead of abandoning the hypothesis that the MMR vaccine somehow causes autism, adherents cling all the more tightly to it, claim all the contradicting data were all a plot by the government and pharmaceutical companies to discredit Wakefield and suppress The Truth About Vaccines; and even go so far as to circulate “We support Dr. Andrew Wakefield” petitions around the Internet. Meanwhile, concerned that a TV special report by NBC’s Dateline would show Wakefield in an unfavorable light, the anti-vaccine propagandists at Age of Autism preemptively launched a strike in the form of many of the fawning posts about Wakefield and attacks on Brian Deer they’ve published over the last couple of years. Thus is the cognitive dissonance that must occur as each new revelation about Wakefield’s incompetence, conflicts of interest, and scientific fraud leaks out. In fact, this is the usual M.O. when it comes to science looking at the claims of any pseudoscientist, be it Hulda Clark, Mark and David Geier, Tullio Simoncini, or whoever the woo-meister du jour is.
Finally, I think it’s worth looking at what the authors concluded in this study:
The main theoretical implication of our research is that “knowledge” as measured on surveys is partly a by-product of the attempt to resolve the social psychological problem of cognitive dissonance. The practical implication of this is that, although scholars have shown a correlation between the perception of links between Iraq and Al Qaeda and support for the war in Iraq, we cannot conclude from this correlation that misinformation led to support for the war. Rather, for at least some respondents, the sequence was the other way around: support for the war led to a search for a justification for it, which led to the misperception of ties between Iraq and 9/11. This suggests a mechanism through which motivated reasoning may be strongest when the stakes are highest . It is precisely because the stakes of going to war are so high that some of our respondents were willing to believe that “there must be a reason.”
In other words, the stronger the emotion behind the belief, the more likely a person is to fall into the trap of using cognitive errors to justify that belief. The key phrase is in the title of the article and in the conclusion, and that phrase is “there must be a reason.” Think about it and how often we hear that sort of a statement in the context of topics relevant to SBM. For example, “there must be a reason” that:
    • my child has autism (it’s the vaccines).
    • there are so many children with autism (it’s the vaccines).
    • there is not yet a cure for cancer (big pharma’s holding out on us to protect its profits).
    • my back pain got better (it must be the acupuncture, not placebo)
    • I rejected chemotherapy for my breast cancer and I’m still alive (chemotherapy is useless and “natural healing” is better).
The list goes on, and all of these are very emotionally-charged topics. After all, in the case of the vaccine-autism belief, what could be more emotional than the bond of a parent to her child, for example? This study doesn’t really break any major new ground, but it does remind me of what I’ve known, namely that, as Richard Feynman once famously said, “The first principle is that you must not fool yourself – and you are the easiest person to fool.” Besides all the ways we can fool ourselves listed by Harriet Hall, trying to minimize cognitive dissonance and using inferred justification are but two more.
It also make me wonder about two things. First, is there a way of taking advantage of these psychological mechanisms to persuade people who hold pseudoscientific views to accept science? In other words, can cognitive dissonance be reduced in a way that doesn’t require a person to reject science and cling ever more tenaciously to pseudoscience and invent conspiracy theories? Second, what beliefs do I hold that are more akin to inferred reasoning or strategies to reduce cognitive dissonance than beliefs based on strong science and firm evidence? I’m just as human as any of the participants in this study. Indeed, any skeptic who thinks he or she is not just as prone to such errors in thinking is not a skeptic but suffering from self-delusion. The only difference between skeptics and non-skeptics, scientists and nonscientists, in this regard is that skeptics try to make themselves aware of how human thinking can go wrong and then act preemptively to try to keep those normal human cognitive quirks from leading them astray. Indeed, guarding against these normal human failings when it comes to making conclusions about the natural world is the very reason we need science and why we need to base our medicine on science.
REFERENCE:
1. Prasad, M., Perrin, A., Bezila, K., Hoffman, S., Kindleberger, K., Manturuk, K., & Powers, A. (2009). “There Must Be a Reason”: Osama, Saddam, and Inferred Justification Sociological Inquiry, 79 (2), 142-162 DOI: 10.1111/j.1475-682X.2009.00280.x

The perils and pitfalls of “doing your own research” about COVID-19 (or any other science)
Ethan Siegel at Forbes argues that you “must not ‘do your own research.'” While the title grates, Siegel is correct that most of us are not really capable of “doing our own research” about most scientific and medical questions because we lack the necessary background. We must therefore be humble and be very, very careful about “doing our own research.”
David Gorski on August 3, 2020
Link

“I’ve done my own research.”
“Do your own research.”
How many times have you heard various antivaxxers, cranks, advocates of pseudoscience, and conspiracy theorists repeat these phrases, or variants thereof? In medicine, advocates of what I like to call pseudomedicine—a category that encompasses antivaxxers, COVID-19 denialists and conspiracy theorists, cancer quacks, and all manner of other quacks—are particularly prone to claim that they’ve “done their research” about, for instance, vaccines, and that’s why they think the MMR vaccine causes autism and that vaccines cause sudden infant death syndrome (SIDS), autoimmune diseases, and all manner of other diseases (and, oh, by the way, their “research” has told them that vaccines don’t protect against disease and “natural immunity is better,” too).
Of course, “doing one’s own research” and then “making up one’s own mind” makes perfect sense when it comes to, for example, choosing a place to live, buying a car, picking a smartphone, and any of a number of decisions we make in our day-to-day lives, although it should be noted that even those decisions are not necessarily so straightforward or easy to research. When it comes to science, the fact is that the vast majority of us are not capable of “doing our own research”. I started thinking about this question again with respect to science-based medicine (and science in general), thanks to an article that bubbled up on social media late last week by former ScienceBlogs blogger Ethan Siegel (who now writes for Forbes), entitled “You Must Not ‘Do Your Own Research’ When It Comes To Science“.
On “doing your own research”
Before I go on, let me just take a moment to observe that the title of his article is absolutely atrocious and doesn’t quite align with what the article actually says. Nowhere does Siegel tell readers that they “must not do their own research.” It’s a title that seems designed to be clickbaity and to inflame, rather than educate. I’m going to guess that Siegel probably didn’t come up with this title, but rather some Forbes editor looking to make the article sound as controversial as possible. Mission accomplished, as you will see, but at the cost of misunderstanding what the article is about. Moreover, I don’t totally agree with everything in the article (that would be a rarity about any article!), but overall it’s a good summation of the perils and pitfalls of lay people “doing their own research” about medicine and science and why lay people, for the most part, are incapable of correctly “doing their own research” on matters of science.
There are few skeptics, if any, who would disagree with Siegel’s introduction, for instance:
“Research both sides and make up your own mind.” It’s simple, straightforward, common sense advice. And when it comes to issues like vaccinations, climate change, and the novel coronavirus SARS-CoV-2, it can be dangerous, destructive, and even deadly. The techniques that most of us use to navigate most of our decisions in life — gathering information, evaluating it based on what we know, and choosing a course of action — can lead to spectacular failures when it comes to a scientific matter.
The reason is simple: most of us, even those of us who are scientists ourselves, lack the relevant scientific expertise needed to adequately evaluate that research on our own. In our own fields, we are aware of the full suite of data, of how those puzzle pieces fit together, and what the frontiers of our knowledge is. When laypersons espouse opinions on those matters, it’s immediately clear to us where the gaps in their understanding are and where they’ve misled themselves in their reasoning. When they take up the arguments of a contrarian scientist, we recognize what they’re overlooking, misinterpreting, or omitting. Unless we start valuing the actual expertise that legitimate experts have spent lifetimes developing, “doing our own research” could lead to immeasurable, unnecessary suffering.
Anyone with expertise who dips their toes into deconstructing pseudoscientific or crank claims regarding issues about which they are deeply knowledgeable will instantly realize that one of the hallmarks of pseudoscience and conspiracy theories is the cherry picking of studies, data, facts, and observations. The reason is simple. The people espousing pseudoscience tend not to look at the evidence base and then make their conclusions fit the evidence. Rather, they start with a conclusion and then go looking for facts, observations, and studies that support that conclusion, ignoring context and, often, uncertainty. It’s known as motivated reasoning, in which a bias towards a conclusion that conforms to what a person already believes leads that person to overvalue information that supports that belief and undervalue disconfirmatory information.
We all engage in motivated reasoning to one degree or another; it’s human nature. One of the main differences between most people and scientists and skeptics is that scientists and skeptics know motivated reasoning exists and make an active effort not to engage in it. It’s not easy, either, as evidenced by the lack of difficulty I have in thinking of scientists (and, yes, some skeptics) who have come to conclusions that appear to be based more in motivated reasoning than on an actual evaluation of the state of existing evidence. It’s easy to name the scientists who have let motivated reasoning guide them down the path of pseudoscience and quackery. All I have to do is to recount the list of physicians and scientists who have gone antivaccine (or become sympathetic to antivaccine viewpoints): Andrew Wakefield, Bob Sears, Kelly Brogan, Paul Thomas, James Lyons-Weiler, Christopher Shaw, Christopher Exley, Judy Mikovits, Theresa Deisher, and many others.
Siegel describes this process rather well:
There’s an old saying that I’ve grown quite fond of recently: you can’t reason someone out of a position they didn’t reason themselves into. When most of us “research” an issue, what we are actually doing is:
    • formulating an initial opinion the first time we hear about something,
    • evaluating everything we encounter after that through that lens of our gut instinct,
    • finding reasons to think positively about the portions of the narrative that support or justify our initial opinion,
    • and finding reasons to discount or otherwise dismiss the portions that detract from it.
Of course, that’s not what we think we’re doing. We think of ourselves as the heroes of our stories: cutting through misinformation and digging up the real truth on the matter. We think that, just by applying our brainpower and our critical reasoning skills, we can discern whose expert opinions are trustworthy and responsible. We think that we can see through who’s a charlatan and a fraud, and we can tell what’s safe and effective from what’s dangerous and ineffective.
Precisely. Many, if not most, of us have a far higher opinion of our critical thinking abilities than is actually warranted. (Some might argue that this description might also apply to me, and so it might, at least for some topics. One always has to take such a possibility under consideration and keep it in the back of one’s mind.) Many, if not most, of us also have a far higher opinion of our knowledge base than is actually warranted. Many, if not most, of us have a far greater confidence in our ability to spot grifters, cranks, pseudoscientists, and charlatans than we, in fact, possess, just as Siegel observes.
The trappings of expertise
This is, of course, exactly why advocates of quackery and pseudoscience so blatantly embrace the trappings of expertise. That’s why Andrew Wakefield, for instance, touts his credentials as a physician. (Never mind that the British General Medical Council yanked his medical license a decade ago) It’s why the deniers of the severity of the COVID-19 pandemic and those who embrace hydroxychloroquine as a COVID-19 cure went full astroturf and produced a video by “America’s Frontline Doctors” claiming that hydroxychloroquine cures COVID-19 and that, because of that, we should reopen schools without concern that doing so might facilitate the spread of coronavirus and not bother with all those pesky mitigation measures like wearing masks and engaging in social distancing. After all, if there’s a cure, why bother with all the difficult methods of slowing the spread of the virus? Unfortunately, the viral video and was viewed tens of millions of times before social media companies removed it from their platforms. Worse, people believed it.
Taking a starring role in this rogue’s gallery of quacks was a Houston doctor and religious minister named Dr. Stella Immanuel:
A Houston doctor who praises hydroxychloroquine and says that face masks aren’t necessary to stop transmission of the highly contagious coronavirus has become a star on the right-wing internet, garnering tens of millions of views on Facebook on Monday alone. Donald Trump Jr. declared the video of Stella Immanuel a “must watch,” while Donald Trump himself retweeted the video.
Before Trump and his supporters embrace Immanuel’s medical expertise, though, they should consider other medical claims Immanuel has made—including those about alien DNA and the physical effects of having sex with witches and demons in your dreams.
Immanuel, a pediatrician and a religious minister, has a history of making bizarre claims about medical topics and other issues. She has often claimed that gynecological problems like cysts and endometriosis are in fact caused by people having sex in their dreams with demons and witches.
She alleges alien DNA is currently used in medical treatments, and that scientists are cooking up a vaccine to prevent people from being religious. And, despite appearing in Washington, D.C. to lobby Congress on Monday, she has said that the government is run in part not by humans but by “reptilians” and other aliens.
In addition, there were several other doctors, including an ophthalmologist and founder of a cryptocurrency company named Dr. James Todaro; a emergency medicine doctor named Dr. Simone Gold known for promoting hydroxychloroquine; another emergency medicine doctor named Dr. Dan Erickson, co-owner of a chain of urgent care centers in Bakersfield, CA who made a name for splash in April with a poorly conceived and carried out “study” full of bad epidemiology that concluded that COVID-19 was far more widespread than thought and that the infection fatality rate was far lower than thought; Dr. Joseph Ladapo, an associate professor of medicine at UCLA and an ardent proponent for the “open up the economy” message who has downplayed the lethality of COVID-19; and Dr. Robert C. Hamilton, a Santa Monica pediatrician who argued for reopening the schools in the fall based on cherry picked studies and information and also made his political orientation very clear with a gratuitous swipe at teachers’ unions in his part of the video (Gizmodo later reported on a letter he wrote in the 1990s expressing anti-homosexual bigotry); and Dr. Richard Urso, an ophthalmologist who has been touting hydroxychloroquine for COVID-19 and claims never to have seen a patient with a heart issue attributable to the drug.
Unsurprisingly, MedPage Today reported that there’s no evidence that any of the doctors in the video got near the “COVID front lines”. Over several days last week, the claims of these doctors were thoroughly refuted, in particular the claim that hydroxychloroquine is, in essence, a cure for COVID-19 that eliminates the need for social distancing and masks to prevent the spread of coronavirus. In fact, more and more, the evidence is trending strongly in the direction that hydroxychloroquine doesn’t work.
What this incident demonstrated very clearly is that, not only are most people unable to evaluate medical claims, but there are a lot of actual physicians out there who are similarly unable to evaluate certain medical claims. It is very telling that these physicians consisted of ophthalmologists, emergency medicine doctors, and doctors who don’t actually have any expertise in virology, epidemiology, clinical trials, or the treatment of COVID-19. This is not a knock on ophthalmologists, emergency medicine doctors, etc., but rather an observation I use to make a point. Even experts in medicine frequently go astray when they wander outside of their field of expertise, particularly when the belief that they want to accept is linked somehow with their sense of self. In this case, it was the politics. These physicians all support President Trump, and President Trump has promoted hydroxychloroquine, downplayed the seriousness of COVID-19, and strongly pushed for the schools reopening, even when there is still doubt about whether it is safe to do so in many areas where the virus is still running rampant.
Basically, like any lay person, when faced with beliefs that they wanted to embrace, “America’s Frontline Doctors” engaged in motivated reasoning and sought out observations, evidence, and cherry-picked studies to give them a reason to support the belief, regardless of whether science actually did support the belief or not. Worse, they very intentionally used their status as physicians to promote those beliefs and persuade lay people to believe them, too. It worked to some extent, as well, although it would have worked a lot better if “America’s Frontline Doctors” had left Dr. Immanuel out of their membership. Her history of claims involving alien DNA and that gynecological problems like cysts and endometriosis are in fact caused by people having sex in their dreams with demons and witches led the hashtag #DemonSperm to trend on Twitter and provided an easy approach to debunking and discrediting her and, thus, the entire group. (It didn’t help that she called doctors doubting hydroxychloroquine fake doctors and the negative studies of the drug “fake studies,” all while accusing those same doctors of being like the “good Germans, the good Nazis” who “watched Jews get killed” and didn’t speak up.) Indeed, it wasn’t long before Dr. Hamilton sent out an open letter to the parents of his patients disavowing Dr. Immanuel’s hydroxychloroquine claims and the claims of other doctors that facemasks are of “no value” in combatting the spread of COVID-19.
Not “one must not do,” but “one must be humble”
Getting back to whether one “must not” do one’s own research, Siegel includes examples other science whose conclusions are not controversial among scientists but are ideologically or politically contentious. These include water fluoridation, human-caused climate change, vaccines, and many of the conclusions regarding COVID-19, correctly noting that, when it comes to evaluating most scientific claims:
Except, for almost all of us, we can’t. Even those of us with excellent critical thinking skills and lots of experience trying to dig up the truth behind a variety of claims are lacking one important asset: the scientific expertise necessary to understand any finds or claims in the context of the full state of knowledge of your field. It’s part of why scientific consensus is so remarkably valuable: it only exists when the overwhelming majority of qualified professionals all hold the same consistent professional opinion. It truly is one of the most important and valuable types of expertise that humanity has ever developed.
But only if we listen to it. It’s absolutely foolish to think that you, a non-expert who lacks the very scientific expertise necessary to evaluate the claims of experts, are going to do a better job than the actual, bona fide experts of separating truth from fiction or fraud. When we “do the research for ourselves,” we almost always wind up digging in deeper to our own knee-jerk positions, rather than deferring to the professional opinions of the consensus of experts.
This is largely true, but I do have a little quibble that I’ll get to in a moment.
First, though, in terms of “doing your own research”:
If you “do your own research,” you can no doubt find innumerable websites, social media accounts, and even a handful of medical professionals who are sharing opinions that confirm whatever your preconceived notions about COVID-19 are. However, do not fool yourself: you are not doing research. You are seeking information to confirm your own biases and discredit any contrary opinions. Each time you do this, you exemplify the problem of anti-science bias that Dr. Fauci warned about in June:
“If you go by the evidence and by the data, you’re speaking the truth and it’s amazing sometimes, the denial there is. It’s the same thing that gets people who are anti-vaxxers – who don’t want people to get vaccinated, even though the data clearly indicate the safety of vaccines. That’s really a problem.”
That is, of course, the problem today. There is a veritable tsunami of misinformation about science out there on social (and, truth be told, old) media. Some of it is intentional disinformation (e.g., antivaccine misinformation to sell products to treat autism or denial of human-caused climate change in order to protect the fossil fuel industry); much of it is ideological (e.g., the promotion of hydroxychloroquine for COVID-19 by Trump supporters); and some of it is religious (e.g., the denial of evolution by natural selection as the major driving force producing the diversity of life). All of it appeals to different people for one reason for another, and all of it can be justified by its believers using motivated reasoning.
There once was a time when it was possible for people without formal education in science to make observations about the universe and formulate them into laws and hypotheses that characterize reality. That time ended a long time. The reason is that science builds on what was discovered. The more it builds, the more background information there is that has to be mastered in order to be able to make useful contributions. Although there can be lots of controversy in science, certain fundamental things are agreed upon because overwhelming evidence has led scientists to provisionally accept them as correct. For instance, you can’t suddenly posit a “theory” that says that atoms aren’t made up of protons, neutrons, and electrons, because there is a massive body of evidence that has led to a scientific consensus that they are, in fact, made up of such particles, whatever scientists choose to name them. At least, you can’t do it and have scientists take you seriously unless you can produce evidence that is at least compelling enough to call such well-established science into doubt. Cranks don’t acknowledge this and, through arrogance, think that they alone are able to see what all of science isn’t. As a result, they tend to be upset that science doesn’t recognize their apparent genius.
Many years ago, I once examined the contention that a “real skeptic always sides with the scientific consensus.” As I noted at the time, in matters of science it is undoubtedly true that the scientific consensus is always the best place to start when evaluating unfamiliar issues. While it is certainly possible that a given scientific consensus regarding an issue can be wrong in almost any area, it nonetheless almost always represents the best current scientific understanding. It is also correct that legitimate authority matters. I emphasize the word “legitimate” because in pseudoscience arguments from authority are common, but rarely is the authority relevant to the point being argued. Often it’s not even legitimate, as in when anti-vaccine activists point to Andrew Wakefield’s work as justification for their claims that vaccines cause autism and other conditions. I also noted that not all scientific consensuses are created equal because, in different fields the strength of scientific consensus can vary quite markedly depending upon the topic or even the subtopic within the topic. For example, the scientific consensus supporting the theory of evolution, particularly common descent, is exceedingly strong. It’s one of the strongest of all scientific consensuses, arguably the strongest. Similarly, the consensus that natural selection is a major driving force behind evolution is very nearly as strong. However, as the discussion devolves into more detailed areas, inevitably the consensus weakens. Eventually, subsidiary areas of a discipline are reached where the consensus is weak or where there is no consensus, such as what the function of “junk DNA” is, whether it is subject to natural selection, and if so how much. (Real evolutionary biologists could probably come up with a better example.) These sorts of questions are often at the cutting edge of scientific knowledge, and it is not always easy to recognize what they are. It is also these issues at the edge of our knowledge that are attacked as proxies for the much more strongly supported core theory.
This brings me back to COVID-19. What makes scientific conclusions about COVID-19 somewhat different than conclusions about vaccines is that the pandemic is new, having only been going on since the disease and virus were first recognized in China in late 2019, and the science is rapidly evolving. That makes it a particularly ripe area for cranks to promote bad science and pseudoscience, particularly given that they can easily invoke the “science was wrong before” trope in real time as new findings come in. However, there are several conclusions that are now pretty firm, including that masks work to slow the spread of coronavirus; that the virus spreads through respiratory droplets, particularly in enclosed spaces; that social distancing works. To that I add that hydroxychloroquine is almost certainly ineffective against COVID-19. It’s possible to challenge these conclusions, but if you do so, you’d better have strong evidence.
In the end, the way to judge claims that go against the current scientific consensus boils down at least as much to tactics and how evidence is used to support such contrarian arguments. Scientific skepticism looks at the totality of evidence and evaluates each piece of it for its quality. In contrast, cranks are very selective about the data they choose to present, often vastly overselling its quality and vastly exaggerating flaws in current theory, in turn vastly overestimating their own knowledge of a subject and underestimating that of experts. In medicine in particular, denialists frequently emphasize anecdotes over epidemiology, clinical trials, and science. They also tend to leap to confuse correlation with causation. Similarly, crankery, denialism, pseudoskepticism (or whatever you want to call it) tends, either intentionally through ideology or unintentionally through an ignorance of the scientific method, to conflate and/or confuse emotional, nonscientific, and/or ideological arguments with scientific arguments. This is not to say that scientists and skeptics and supporters of SBM are free from their own biases, whether ideological or simply a desired result that they hope to find. Far from it. However, skepticism means applying the scientific method to claims, and whatever its faults, the scientific method is the best method thus far devised to minimize these biases.
As scientists, the reason we use the scientific method is not because we consider ourselves superior to the cranks, but rather because we recognize that we are human too and thus just as prone to falling into the same traps as they. As Richard Feynman once famously said, “The first principle is that you must not fool yourself—and you are the easiest person to fool. So you have to be very careful about that. After you’ve not fooled yourself, it’s easy not to fool other scientists. You just have to be honest in a conventional way after that.” The scientific method is, above all, a methodology by which scientists try to avoid fooling themselves. Skeptics cross the line dividing skepticism and denialism and quacks the line between science and quackery when they forget that. Doubting a scientific consensus is not in and of itself the mark of the crank. It’s how and why that skepticism exists that distinguishes crankery from genuine scientific skepticism.
The problem with “doing your own research” is that rarely does a lay person (or even a physician or scientist venturing too far outside of his area of expertise) have the background knowledge and skillset to be confident of avoiding crossing that line, whether intentionally or not. It’s not so much that you “must not do your own research.” It’s that you really need to understand that you probably can’t “do your own research” and that the conclusions you reach “doing your own research” are highly likely to be more in line with your prior beliefs than scientifically correct.
As Siegel puts it:
But that requires a kind of transformation within yourself. It means that you need to be humble, and admit that you, yourself, lack the necessary expertise to evaluate the science before you. It means that you need to be brave enough to turn to the consensus of scientific experts and ask, legitimately, what we know at the present stage. And it means you need to be open-minded enough to understand that your preconceptions are quite likely to be wrong in some, many, or possibly even all ways.
Humility! What a concept!

Quoth Myrna Mantaring: “US government data” confirms a “143,233% increase in cancer cases due to COVID vaccination”? I answer with a plea for math-based reality checks.
Myrna Mattaring, a retired scientist who worked in diagnostic labs, claims that COVID-19 vaccines caused a 1432% increase in cancer cases, a clearly impossible claim. Here I make a plea for examining such claims, including a much more famous and accepted one, with basic math.
David Gorski on September 3, 2024
Link

If there’s a lesson that I’ve learned about the claims of cranks, quacks, and pseudoscientists, it’s that too much is never enough in terms of their hyperbolic claims. For example, with respect to COVID-19 vaccines and cancer, it wasn’t enough just to falsely claim that COVID-19 vaccines cause cancer. Oh, no. They had to cause “turbo cancers,” cancers so much nastier, fatal, and prone to appearing in younger people than regular boring, run-of-the-mill cancers. Nor is it enough to say that the vaccines have caused the incidence of cancer merely to have doubled or tripled—or even quadrupled—any of which, if true and provable with statistics and epidemiology linking vaccination to cancers, would be alarming enough. That’s not enough for cranks. Oh, no. Behold the latest claim from Myrna Mantaring, MT, MCI, MS, MBA, CLSp MB (ASCP), who “has more than 50 years of experience in the fields of Diagnostic and Research Laboratory Technology, Microbiology, Immunology, and Molecular Biology” (but apparently none in epidemiology), who has appeared with antivaxxer Polly Tommey (buddy of Andrew Wakefield) on CHD.TV to claim that there has been a “143,233% increase in cancers due to COVID vaccination” in the US:
"The increase in cancer, according to the latest report, the US government data confirms [a] 143,233% increase in cancer cases due to COVID vaccination. That's unreal. But it's true."Myrna Mantaring, MT, MCI, MS, MBA, CLSp MB (ASCP), who has more than 50 years of experience in… pic.twitter.com/kzDcuHDb9c
— Sense Receptor (@SenseReceptor) August 18, 2024
“US government data”? Which “US government data”?
Amusingly, even some antivaxxers are responding with “WTF?”:
I don't think the increase is that high.
— My Dog Is Fat (Fat Dog Acceptance Movement) (@CypressXMX) August 18, 2024
“I don’t think the increase is that high” is a hilarious understatement.
Interestingly enough, the video of Polly Tommey interviewing Myrna Mantaring was first posted on CHD.TV on August 14 under The People’s Study but it didn’t appear to start gaining much traction until last week, when The Lioness of Judah posted a link to the video on their popular conspiracy Substack, Exposing the Darkness, having shown up on—where else?—Rumble.
The money quote that all the sources amplifying this video cite is this:
The increase in cancer, according to the latest report, the US government data confirms [a] 143,233% increase in cancer cases due to COVID vaccination. That’s unreal. But it’s true.
Big, if true.
Actually, Mantaring was right the first time, when she categorized it as “unreal.” But who, pray tell, is Myrna Mantaring? Her history, as described as follows. From 2005 to 2015 she served as a military hospital’s health science administrator and research-technology laboratory manager. (Note that the hospital is not named.) From 2015 until her retirement in 2021, she also worked as a clinical-diagnostic laboratory manager for the same hospital’s department of nephrology. Again, these are not really qualifications that I would associate with the ability to be able to parse and analyze “US government data” regarding COVID-19 vaccination and cancer, much less to conclude that COVID vaccines had caused a 1,423.33-fold increase in cancer cases when even a quack like William “turbo cancer” Makis doesn’t make such a claim.
So why write about a fringe claim that is clearly absolute bollocks (as my British friends would put it)? I would argue that it’s worth my attention for the same reason that writing about Dr. Marty Makary’s bogus “estimate” that medical errors are the third leading cause of death in the US, namely to examine such factoids under the lens of critical thinking and basic mathematical literacy. The main difference between the two is that Makary’s estimate became so widely believed that I keep seeing it pop up, mentioned as an accepted factoid, even in ostensibly evidence- and science-based sources, like John Oliver’s This Week Tonight. It doesn’t even matter that, like Mantaring’s estimate, it doesn’t even pass the “smell test” in terms of being plausible from a strictly mathematical standpoint, although, admittedly, it is more plausible than Mantaring by a lot.
Here’s what I mean.
Makary’s estimate in 2016 was that that over 251,000 people a year were dying in hospitals as a result of medical errors. According to the CDC, in 2010 only 715,000 of US deaths occurred in hospitals, implying that, if Makary’s estimate were accurate, then 35% of inpatient deaths are due to medical errors. It was worse than that, though, because Makary’s upper estimate was 400,000 inpatient deaths due to medical error. That was 56%—yes, 56%—of all inpatient deaths! As I said at the time It wasn’t anywhere near plausible that somewhere between one-third to over one-half of all inpatient deaths in the US were due to medical error. It just wasn’t, unless you’re a conspiracy theorist of the type who believes that hospitals were intentionally killing COVID-19 patients through their protocols. Of course, Makary’s estimate, based as it was on unjustified extrapolation from small studies that used an overly broad definition of death by medical error, was just low enough to seem plausible if you didn’t look too closely at the numbers and didn’t compare them to known statistics.
Now, let’s do the same thing with Myrna Mantaring’s estimates. To do that, I head over to the CDC’s website and look at the incidence of new cancer cases by year:
There are a couple of things right here that we can look at. First, note that the trend in new cancer cases per 100,000 population is actually slowly declining. Now, let’s look at the numbers for the year 2020, given that COVID-19 vaccines didn’t receive emergency use authorization (EUA) until December of that year and mass vaccination didn’t really take off until early 2021. According to the accompanying data table, the models rate in 2021 (which I use because the actual rate was lower due to disruptions in screening and patient care due to the pandemic in 2020) was 447.5 per 100,000. Now, if Mantaring were correct and the incidence of new cancer cases increased 143,233% due to COVID-19 vaccines, it’s a simple matter to see what that would look like by multiplying 447.5 by the 1,432.33-fold increase claimed by Mantaring. If you do that, you’ll quickly come up with an estimate for incidence of new cancer diagnoses per year of 640, 968 per 100,000, or more than 6.4 times what is possible. That Mantaring didn’t do that sort of basic plausibility check on her number tells me all I need to know about her.
Moreover, even if you accept the idea of “turbo cancer,” in which COVID-19 vaccines are supposedly so carcinogenic that they can cause cancer faster than ionizing radiation (two to ten years, depending on the specific malignancy), you would expect to have started seeing the trend in the numbers by 2022 at least, and the trend couldn’t be the least bit subtle if we were going to achieve the mass death from cancer estimated by Mantaring by this year. Basically, like Makary’s estimate, Mantaring’s estimate doesn’t pass the basic smell test, but even more obviously. Truly, mathematical literacy is not a characteristic of cranks, although at least Makary didn’t produce a number higher than was physically even possible.
Watching the video, I looked for an explanation for how Mantaring came up with her estimate. I was amused by her introduction. Tommey wanted to talk about Mantaring’s sister, but Mantaring wanted to list her credentials, none of which were particularly relevant to her overarching claim regarding cancer. She also pointedly mentioned that when the pandemic hit in March 2020 she was considered an “essential employee” and came to work every day, which is also fairly irrelevant given that it would be expected that laboratory personnel responsible for doing lab tests in a hospital can’t do their jobs remotely. But she did start “following all news, seminars, symposia, and investigations related to the pandemic, the early treatment, the development and approval of the gene-based vaccine, as well as the CDC reporting system.” Impressive, if true, given that the sheer volume of “news, seminars, symposia, and investigations” described would have been beyond the ability of one person to follow them all while doing her job, which more and more sounds like more of a laboratory technician than an investigative scientist.
Of course, because Mantaring performed PCR diagnostics of viral diseases, she claims that she was immediately able to “recognize the government’s continuing fraud and deception in their COVID-19 testing and vaccine policies,” which she characterized as “crimes against humanity” and part of the “World Economic Forum’s diabolical new world agenda.” She also mentions how “fetal cell lines” were used in the manufacture of the vaccines, a longstanding antivax trope that seeks to weaponize religious objections about abortion to sow disgust about vaccines, while going on about the “limited safety data” and supposedly rushed testing—and this all in the first three minutes of the video!
I wasn’t really interested in going over the same old, same old antivax tropes about COVID-19 vaccines, nearly all of which Mantaring hit at one time or another, along with “early treatment protocols” about hydroxychloroquine and ivermectin, which she urged her family and friends not to take the vaccine but to use these repurposed drugs, which do not work. Unsurprisingly, she states that the reason she retired in 2021 was to avoid mandatory COVID-19 vaccination at her job. That is why I basically listened to the video in the background, waiting for something—anything—describing how she had come up with her ridiculously implausible estimate for such a massive increase in cancer incidence being linked to the COVID-19 vaccines.
I did learn, however, that Mantaring’s sister in Toronto was diagnosed with sarcomatoid urothelial carcinoma, a rare and aggressive cancer of the bladder, “after being fully vaccinated and boostered as mandated by the vaccine passport per the Trudeau government.” Sadly, she couldn’t tolerate the preoperative chemotherapy and was scheduled for surgery in April 2022, at which point it had invaded the uterus and surrounding bowel, requiring more extensive surgery. Eventually it spread to the liver, and her sister died from her disease, seven months from diagnosis. While this is a tragic story, it is not unusual for a cancer like this one, which is aggressive. After this, Mantaring claims that her sister’s story confirmed for her what she had been hearing from people in her circle about cancer in vaccinated people, leading her to go on and on about how spike protein supposedly causes cancer, facilitated by its also causing “vaccine-acquired immune deficiency,” a nonexistent syndrome that antivaxxers like to call VAIDS.
So I listened to the video, wondering if there was any more than the quotes about cancer from COVID vaccines that had been highlighted by The Lioness of Judah and various antivaxxers on X, the hellsite formerly known as Twitter, and other social media, other than this in response to a question about Tommey why the public “fell in line with” and “obeyed” government vaccine mandates “without thinking” (of course—sheeple!):
Because they trusted the government. They would not think that the government will lie to them. And it’s also a belief that a sophisticated form of medicine from big pharma, no less, big names—Pfizer, AstraZeneca, J&J—you would think, they would have no second thoughts. But what triggered my suspicion was, coming from an immunology background and working in an immunization clinic, we don’t do this. We don’t push out a vaccine with less than a year of trials.
Actually, in a pandemic, we do, and the testing was adequate. Elsewhere, she asks:
How can a government people trust lie to them?
Good question, but few are the people who trust the government never to lie, particularly in our current political climate.
At this point, I was getting tired; so I’m just going to come around to the punchline. Myrna Mantaring doesn’t present any evidence to support her claim that as a result of COVID-19 vaccination cancer cases have increased by 143,233%, other than referring to unnamed and undescribed US government data. Her entire argument on this point boils down to basically, “Trust me, bro.” No, seriously. She doesn’t even show graphs or charts. Even cranks as cranky as William Makis show graphs and charts. They might be poorly conceived, designed, and analyzed graphs and charts, but they are at least an attempt to show evidence. From Mantaring, we get nothing.
It’s worse than that. With a little help from a reader, I was made aware of the origin of the “143,233% increase” in cancer claim, and it’s about what you might have predicted. First, it’s not a new figure. In fact, it’s older than even I had suspected. It comes from a conspiracy website called The Exposé, started in 2020 by a British welder named Jonathan Allen-Walker, and an article published there in August 2022 entitled U.S. Government data confirms a 143,233% increase in Cancer cases due to COVID Vaccination. The analysis involved the Vaccine Adverse Events Reporting System (VAERS), because of course it did. Basically, the anonymous author of the article examined cancer-related reports to VAERS related to flu vaccines administered from 2008 to 2020 and COVID-19 vaccines administered between December 2020 and August 2022, and came up with this graph:
I’ve explained more times than I can remember why VAERS is not reliable when it comes to estimating population prevalence of a given condition or for linking that condition to vaccines, given the base rate fallacy and thus don’t know the base rate of the given condition to compare the VAERS data to. Moreover, after the release of COVID-19 vaccines, for the first time ever, the CDC publicized VAERS and encouraged reporting adverse events to it, which, to the surprise of no one familiar with VAERS, led to a huge spike in reports of adverse events related to COVID-19 vaccines. Again, this number fails a basic plausibility test, and only the mathematically illiterate would take such an estimate seriously.
As I reach the end, again one might wonder: Why did I bother? Myrna Mantaring is so obviously off the deep end that it seems pointless to discuss a clearly and obviously fantastical estimate for how much COVID-19 vaccines supposedly increase the number of cancer diagnoses. I bothered because, for one thing, to most people it’s not obvious that her estimate is so obviously off the deep end and mathematically impossible. To know that takes background knowledge. For example, just look at some of the responses in The Lioness of Judah’s comments after one commenter suggests that there’s no way the increase is this high, even though that commenter seems clearly to believe that COVID-19 vaccines contribute to cancer:
You get the idea.
In the end, I wrote about this as a basic plea to do a reality check whenever you see a figure that seems implausible or inflated. With Mantaring’s estimate, it’s really easy, which is why I chose it, much the same way that I like to use homeopathy as an example of extreme implausibility. All you have to do is to look up cancer incidence statistics and then see if what she is claiming is even possible. (It isn’t.) For a claim like Marty Makary’s “third leading cause of death” claim about medical errors, it’s trickier. First, you have to parse his claim a bit and realize that it’s only about in-hospital deaths, not all deaths. Then you have to look up the statistics for how many people die in hospitals in the US every year, so that you have something to compare to. Doing that, you will see that, while Makary’s estimate of 250K to 400K deaths due to medical errors in hospitals is possible, it is incredibly implausible that such a high percentage of deaths in hospitals are due to medical errors. If you want to go further, you can then compare Makary’s number to total deaths in the US every year. If you do so, you’ll find that it’s also pretty darned implausible there too, given that at the time there were estimated to be 2.6 million deaths per year in the US; Makary’s estimate would imply that medical errors cause between 10% and 15% of all deaths every year. Even that comparison tells you how implausible Makary’s numbers were; yet almost no one pointed this out, and his “third leading cause of death” factoid became so ubiquitous that even other ostensibly reason-based sources routinely cite it.
Skepticism and critical thinking are your friends, and I hope that this little exercise, even carried out on an estimate from an obvious crank whose argument boils down to “trust me, bro” inspires you to apply such methods to any numbers-based claim that doesn’t sound plausible or correct. Now, if only the press and many of my medical colleagues who still parrot Makary’s figure would do the same thing.

   Nine differences between “us and them,” nine straw men burning
David Gorski
Link
I’ll start this post by admitting right up front: I blatantly stole the idea for the title of this post from Mark Crislip’s most excellently infamous post Nine questions, nine answers. Why? Because I really liked that post and felt like it. Also, there seems to be something about the number nine among anti-vaccine zealots: Nine “questions.” Nine circles of hell.
Nine straw men.
I’m referring to an amazing post that appeared on the anti-vaccine crank blog Age of Autism over the weekend by contributing editor Julie Obradovic entitled The Difference Between You and Me. In this post, Julie describes not one difference, but nine differences, that she perceives between herself (and, apparently, by generalization other parents who have become believers in the myth that vaccines cause autism) and people like SBM contributors and (I hope) the vast majority of our readers, who support science-based medicine, understanding that correlation does not necessarily equal causation and that, most importantly, science not only does not support the belief that vaccines cause autism but provides us with copious evidence that there almost certainly no link between the two. Actually, there are more than nine differences, as Ms. Obradovic packs multiple apparently related differences around each of her nine “differences” and then complains that Alison Singer and, apparently by generalization the rest of us who support SBM and oppose the anti-vaccine movement, misrepresent the reasons why she and her merry band of anti-vaccine activists reject the science that has failed spectacularly to validate their deeply held belief that vaccines cause autism and all sorts of other health consequences. Her post ends up being a collection of straw men constructed to Burning Man size, each of which she then applies a flamethrower of burning nonsense to with self-righteous gusto.
Although no doubt Ms. Obradovic won’t see it that way, the reason I chose her article as an introduction for this post is not to pick on her (although if you look at her other posts on AoA, particularly her equally large city of straw men entitled How to Actually Save the Vaccine Program) she certainly deserves some picking on for her combination of pseudoscience, logical fallacies, and straw men). Rather, it’s because her collection of straw men are highly illustrative of what supporters of SBM have to deal with when dealing with pseudoscience and quackery. Ms. Obradovic’s “nine differences” may be all about vaccine-autism mythology and victimization that those mean and nasty scientists don’t take her beliefs seriously, but they could be about almost any form of non-science-based medicine. If you don’t believe me, do this simple thing. Wherever Ms. Obradovic writes “vaccines,” insert your favorite woo du jour and then channel the all-purpose quackery crank site Whale.to or NaturalNews.com. It doesn’t work for all of them (the part about the government “mandating” vaccines, for instance), but it works for enough of them to show my point.
Another reason why I’m going to discuss Ms. Obradovic’s collection of massive straw men peppered with other logical fallacies is that her attitude is not unique. What she writes demonstrates some key attitudes and belief systems towards science and points out many of the obstacles that those of us who try to promote science over pseudoscience, whatever the field, be it vaccines or any other area of quackery or pseudoscience, must address and overcome.
Straw men on flame with logical fallacies (apologies to Blue Öyster Cult)
Ms. Obradovic appears to be very incensed about a talk that Alison Singer, President of the Autism Science Foundation, arguably the only truly science-based autism charity in existence at the moment, and she uses a talk by Singer that was lambasted by Generation Rescue big macher J.B. Handley in three parts, as her jumping off point:
I am growing increasingly tired of the real reasons there is such controversy regarding vaccines and Autism being misconstrued to make me look pathetic. Alison Singer’s attempted explanation at Yale earlier this month (HERE) is a perfect example.
Contrary to what she suggests, our differences are not due to the internet. They are not due to desperation or the traumatization of having a child with Autism coupled with the need to blame someone. They are not due the media or the anti-establishment-toxic-earth movement. They are not due to the dismissive attitude of society and physicians who for years believed bad parenting was to blame. They are not due to an inability to simply accept clear science. They are not due to lack of an education or ability to think rationally. They are not due to being taken advantage of. They are not due to the cult of celebrity.
Wrong. Sorry. Not even close.
Actually, all of the above are excellent partial explanations for why parents like Ms. Obradovic refuse to accept science and continue to believe that vaccines cause autism. The reason Ms. Obradovic “looks pathetic” is not because some cabal of scientists are trying to make her look pathetic; it’s because she does an excellent job by herself of making pathetic arguments.
As I pointed out above, the rest of Ms. Obradovic’s post is structured as nine descriptions of what scientists and those of us who accept the science that doesn’t support her belief that vaccines cause autism believe, and her responding, “I don’t,” followed by a heapin’ helpin’ of what she does believe. Unlike Mark’s post, I’m not going to cover each and every one of these fallacies one by one in detail the way Mark did. I will note that upon rereading the post I just realized that Ms. Obradovic forgot a #3, disobeying the rule regarding the Holy Hand Grenade of Antioch: “Three shall be the number thou shalt count, and the number of the counting shall be three. Four shalt thou not count, neither count thou two, excepting that thou then proceed to three.” Of course, Ms. Obradovic has two straw men #5; so it all evens out to nine straw men again.
So let’s start with straw man #1:
1. You believe the government (the Department of Health and Human Services) has the legally protected right to research, develop, patent, license, supervise, judge, approve, recommend, mandate, and profit from a product (vaccines) that they produce in partnership with a private entity (the pharmaceutical industry). You further believe they have the right to simultaneously oversee the quality, safety and efficacy of this product, and that they objectively do so. You even further believe that they have the right to fund and conduct studies used to defend their product and policy in a court that they serve as judge and jury over in the event you are harmed by it; and moreover, that if they do find in your favor, they have the right to award you compensation at their discretion using money that was secured by a tax you paid on the product when you purchased it and/or were mandated to use it. And finally, you believe this should be protected by law; that neither the government nor the private entity should be held criminally or financially responsible for negligence in the event it maims or kills you.
I don’t.
See what I mean? Straw man #1 is in reality several straw men prefaced by misrepresentation of Ms. Obradovic’s opinion as fact. The fct is that the government does by law have the power to fund the development of vaccines, purchase them for government-run vaccination programs, and recommend them. Note also how Ms. Obradovic is specifically castigating the federal government (DHHS is a federal, cabinet-level department) when in reality the federal government has relatively little power to mandate vaccines in the civilian sector. (If you don’t believe that, check out how few Americans were vaccinated against H1N1 last year despite a massive government effort to persuade Americans to be vaccinated). It is the state and local governments that set specific vaccine mandates required of children. True, they usually use the recommendations made by the CDC and AAP, but it is not primarily the federal government that “mandates” vaccines.
As for compensation, Ms. Obradovic is referring to the Vaccine Court. While it is true that the government requires that claims against vaccine manufacturers first be adjudicated through the Vaccine Court, as has been described on this blog and elsewhere, the Vaccine Court is actually a complainant-friendly venue, where the Daubert rule for scientific evidence is usually not enforced and it is not necessary to prove negligence. Moreover, for purposes of the Vaccine Court, there are a set of “table injuries,” which are in essence recognized potential complications from vaccines that are automatically compensable. These injuries are simply assumed to have been from vaccines, based on science documenting these as potential complications from vaccines. Also, unlike regular courts, the Vaccine Court will pay attorney’s fees and reasonable expenses even to losing petitioners. A petitioner need only demonstrate that the petition was filed in good faith and that there was a reasonable basis for the claim, the idea being to make it as easy as possible for ordinary citizens to seek compensation for vaccine injuries without incurring huge legal bills or being unable to find a lawyer to represent them on a contingency basis. Indeed, attorneys like Clifford Shoemaker have made quite the cottage industry of bringing claims before Vaccine Court, knowing that their expenses will be paid, win or lose. Finally, if an action fails in Vaccine Court, the parent is perfectly free to pursue it in the regular courts.
What straw man #1 reveals is that, like many supporters of pseudoscience and crankery, Ms. Obradovic views science and the government as being arrayed against her, all in cahoots with big pharma. Whatever the short comings and misbehavior of big pharma, some of which I’ve personally documented right here on this very blog, cranks like Ms. Obradovic go far beyond reasonable concerns about big pharma, as we will see.
On to straw man #2:
2. You believe the only protection the consumer needs to be afforded in the aforementioned situation is trust. People should simply trust that those given such enormous power and protection are honorable, ethical, and responsible human beings with families of their own who would never abuse it or put profit over safety primarily because they are smart, went to prestigious medical institutions, and are at the top of their field. You do trust them. And you trust that there are just too many of them involved to all be bad, somehow making the system safe from corruption based on numbers. This is the one point people rely on to debunk the “conspiracy theorists”.
I don’t.
First of all, trust is earned. So is respect. I don’t care what letters you have after your name. You’re smart? Great. So am I. I’m not impressed, nor am I intimidated. Smart doesn’t mean ethical. And some of the smartest people I know are also those with the least common sense.
Does this sound familiar? It’s very much the same anti-intellectual attitude that J.B. Handley once bragged about. Here’s a hint for Ms. Obradovic. Being “smart” isn’t what matters. “Common sense” isn’t what matters. Understanding and accepting the scientific method and how science works does. Ms. Obradovic honestly seems to believe that the reason the scientific community doesn’t accept her wild beliefs that vaccines cause autism is because of a lack of ethics, plus the government, big pharma, and scientists being all in some grand conspiracy, not because the scientific evidence doesn’t support her belief. Unlike the case for scientists, it never occurs to Ms. Obradovic that she might be wrong or that the reason her belief that vaccines cause autism are not taken seriously by scientists is because, well, she is wrong. But not just wrong, spectacularly and arrogantly wrong about the science. It is the the pure arrogance of ignorance, born of anti-intellectualism.
In brief, I’m not in the least bit impressed by Ms. Obradovic’s trumpeting of her being so “smart.” She has not earned respect in any scientific discussion–quite the contrary. She has proven time and time again that she does not know what she is talking about when it comes to science and that her emotion and distrust of science trump all. Contrary to Ms. Obradovic’s apparent belief that the science behind vaccines is rotten to the core, in actuality it is scientists, not misguided conspiracy mongers like Ms. Obradovic, who point out the shortcomings in the vaccine program.
Next, I’ll skip ahead a bit, because it’s a lot more of the same ranting about the government and the scientific community, and it can all be boiled down to straw man #5-1 (given that there are two straw men #5) anyway:
5. You believe the science funded and conducted by the DHHS, pharmaceutical companies, vaccine patent holders and government witnesses (there exists no widely accepted study without this level of participation and conflict) thus far on the potential role between vaccines and the onset of Autism Spectrum Disorder and other health outcomes (for which they will be held accountable) is objective and adequate as it stands right now in both quantity and quality to dismiss a link between the two.
I don’t.
There is not enough space in this article to explain why, but a detailed explanation can be found through the series of articles I wrote here at Age of Autism on the 14 Studies. I’ve read, analyzed and presented every single study multiple times. What you call clear science, I call crap. And no, I’m not willing to accept crap when it comes to my child.
The problem is, of course, that Ms. Obradovic doesn’t have the background to determine whether a scientific study is “crap” or well-designed, well-executed, and reliable. It is the arrogance of ignorance once again asserting itself. In addition, it is a straw man to claim that we supporters of SBM believe that the science “funded and conducted by the DHHS, pharmaceutical companies, vaccine patent holders and government witnesses” is adequate because there’s so much more than evidence funded by the U.S. government. There’s more to the world than just the United States, you know. There have been many studies not just in the U.S., but in several other countries, including Denmark, Canada, the U.K., Japan, Italy, and elsewhere that have failed to find a link between thimerosal in vaccines and autism or between vaccines and autism. Surely all these countries can’t be in on the conspiracy, can they? A much more accurate way of saying this, without the intentional use of the logical fallacy of poisoning the well, is that the totality of the scientific and clinical evidence, when taken as a whole, does not support a link between either thimerosal in vaccines and autism or between vaccines and autism. In contrast, by mentioning the execrable Fourteen Studies website, Ms. Obradovic demonstrates that what she views as “good science” is any science that reinforces her belief, no matter how biased it is or poorly designed and executed, as Steve Novella, Mark Crislip, and I have all demonstrated in our deconstructions of that particularly misinformation-packed Generation Rescue-sponsored propaganda effort. Let’s just put it this way. Anyone who doesn’t easily see through the distortions and misinformation in the Fourteen Studies website has forfeited any claim to an understanding of scientific studies adequate to make grandiose statements about the validity of existing science, such as what Ms. Obradovic makes in straw man #5-2:
5. You believe everything about Autism is a coincidence: the dramatic rise in incidence; the parallel increase in vaccinations given at the same time; the similarities to mercury poisoning; the ratio of boys to girls; the identification of this new disorder in 1943; the timing of the onset of symptoms; the anecdotal evidence of parents; the original CDC findings; the recovery of children who are treated medically; and more.
I don’t.
Science is rooted in observation, and yet, every observation here listed is casually tossed aside as a cosmic lining up of the stars. There is nothing scientific about calling all of this coincidence and explaining it away with unproven excuses (see your list in the second paragraph)…and crap.
This particular straw man demonstrates a misunderstanding of epidemiology so profound as to be beyond belief. In actuality, Ms. Obradovic’s “observations” are nothing more than the blatantly obvious confusing of correlation with causation. As has been pointed out time and time again, mercury poisoning and autism do not resemble each other that strongly. The “dramatic rise” in autism incidence can be largely (although it is unclear if it can be completely) explained by widening of the diagnostic criteria and diagnostic substitution. Also, an example I’ve used before is the Internet. The rise in Internet use beginning in the early 1990s very closely parallels the rise in autism diagnoses and autism prevalence. Surely, by Ms. Obradovic’s logic, the Internet should be just as plausible as a cause of autism as vaccines.
She’s also dead wrong that the hypothesis that vaccines cause autism has been “casually tossed aside” as coincidence. In fact, scientists have studied extensively vaccine safety, looking for a linkage between vaccines and autism, largely driven by the concerns of mothers like Ms. Obradovic. They haven’t found any. In fact, I can retort that one difference between someone like Ms. Obradovic and someone like me is that she doesn’t understand that correlation does not equal causation and that, when science has failed to find a linkage between two things, when that the most likely explanation for any linkage between the two is coincidence. It’s a really hard concept for most people to accept, particularly when they have an emotional investment in a claim of causation, but it’s true. Confusing correlation with causation, confirmation bias, and a number of other cognitive factors conspire to prevent people from easily accepting that sometimes bad things are a coincidence.
One excellent example came from an article about the H1N1 vaccine using the example of H1N1 vaccination and heart attacks.Given that there are this number of people having heart attacks each and every day, during the months when so many people were being vaccinated against H1N1, it was inevitable that there would be dozens, if not hundreds of coincidences a day in which something bad happens to a person after having the H1N1 vaccine. If you’re one of those people, it will seem all the world as though the vaccine caused the badness to happen. It’s not because these people are stupid or ignorant; it’s because, not knowing the expected rate of these coincidences, most people assume that the rate of coincidence is far lower than it truly is. They assume that the rate is close to zero, that such a coincidence would be rare, but that assumption is wrong when dealing with large numbers.
Yes, that’s one difference between Julie Obradovic and me. I understand that. She doesn’t. She thinks herself to be too “smart” ever to make the mistake of mistaking correlation for causation. One of my favorite movie quotes of all times comes from, of all places, a Dirty Harry movie, specifically, Magnum Force. In it, Dirty Harry Callahan says at one point, “A man’s got to know his limitations,” and at another point, “A good man always knows his limitations.” This applies to women as well as men, and Julie Obradovic doesn’t know her limitations with respect to science. From my perspctive, if Obradovic’s world view were more accurate than mine, if big pharma really did have the power to fake research findings all over the world, I’d have to wonder: Why bother to put all those alleged “toxins” in vaccines? Why not use homeopathic vaccines, something harmless but ineffective, and then make up evidence to make it look as though they work?
I think that, in the end, the difference between Ms. Obradovic and someone like me, a supporter of science-based medicine, is that there is evidence that, if produced, would change my mind about whether or not there is a link between vaccines and autism, and I know what that evidence would have to be right now. All it would take would be a couple of well-designed, well-executed, well-analyzed epidemiological studies showing a strong link between vaccines and autism. Produce those, and I would start to reconsider my position. Or, as Tim Minchin put it so brilliantly about homeopathy in his nine minute beat poem Storm (please be warned that Minchin is fond of the f-word):
Science adjusts its beliefs based on what’s observed
Faith is the denial of observation so that Belief can be preserved.
If you show me
That, say, homeopathy works,
Then I will change my mind
I’ll spin on a fucking dime
I’ll be embarrassed as hell,
But I will run through the streets yelling
It’s a miracle! Take physics and bin it!
Water has memory!
And while it’s memory of a long lost drop of onion juice is Infinite
It somehow forgets all the poo it’s had in it!
You show me that it works and how it works
And when I’ve recovered from the shock
I will take a compass and carve Fancy That! on the side of my cock.
The same goes for me and not just homeopathy, but the belief that vaccines cause autism. I’d be embarrassed as hell for having been wrong, and I might resist changing my mind for a while, but eventually science would win out, and I’d realign my beliefs to conform with science. I would, however, abstain from bringing any sharp instruments anywhere near my genitals, and I’m not sure if I’d go running through the streets yelling, “Vaccines cause autism!” I would, however, write about it right here on SBM, minus the use of the f-word. In contrast, there is clearly no evidence that will ever change Ms. Obradovic’s fanatical belief that vaccines cause autism. Just try asking her if you don’t believe me.
The question that remains is: Why do people like Julie Obradovic refuse to accept the science that shows that vaccines are safe and effective and that they are not associated with autism? I’ve already pointed out one reason: Failure to understand that correlation does not equal causation, coupled with failure to let go of a belief that isn’t supported by science. Obviously, though, that alone is not sufficient to explain the intensity of her reaction.
Next week (or the week after if something comes up that catches my fancy), I’ll consider mechanisms by which we protect irrational beliefs from science.

Why It’s Worthwhile to Debunk Shivambu
Richard Feynman famously said, “The first principle is that you must not fool yourself—and you are the easiest person to fool”. This is the most important lesson skepticism has to offer.
Jonathan Howard on December 17, 2021
Link

What is Shivambu?
An anti-vaccine doctor I trained with, Dr. Kelly Brogan, recently interviewed a man who claims he cured his cancer by drinking his own urine, a practice called Shivambu. Literally everything about this is sad and disgusting. Though she’s retired from the practice of medicine and is now a full-time guru/online marketer, such balderdash is standard fare for Dr. Brogan and her ilk. However, the point of this article is not to debunk Shivambu, but rather to explain why it may be worth someone’s time to do so.
Obviously, few people reading this article believe that drinking urine cures cancer, and those who believe such folly are unlikely to have their mind changed by it. So other than maybe convincing a few people that Shivambu is nonsense, why bother writing about it?
Several intelligent people have argued that sites like Science Based Medicine are essentially a waste of time (here and here). These critics feel that skeptics do little other than “bashing people outside your tribe, who ignore you.” They claim skeptics are simply “preaching to the converted”, and so accomplish nothing beyond making themselves feel superior. They feel that alternative medicine may be worth discussing only after mainstream medicine has fixed all its flaws and world peace is declared.
I don’t claim to speak on anyone’s behalf, but I think that when skeptics discuss something like Shivambu, their main aim often isn’t to debunk that specific nonsense. Rather, I think most skeptics have several larger goals in mind, and discussions of topics like Shivambu are merely a template for them to make their main points.
The lessons of skepticism part 1: People are seriously hurt and killed by alternative medicine all the time
Having said this, one reason to debunk Shivambu is to debunk Shivambu. Critics of skepticism often portray alternative medicine as ridiculous, but ultimately harmless. “What’s the harm if someone believes crystals ward off disease?” they ask.
While many alternative treatments are harmless, many others are not. People can be harmed directly by alternative therapies or because they used them in lieu of evidence-based treatments. For example, one study found that “After a median of 5 years, patients with breast or colorectal cancer were nearly five times as likely to die if they had used an alternative therapy as their initial treatment than if they had received conventional treatment.” As the pandemic has demonstrated, people are seriously hurt and killed by medical misinformation all the time. Just because war kills more people, doesn’t mean we shouldn’t try to save lives in other ways, a flawed line of thought dubbed the fallacy of relative privation.
It’s good when doctors warn people about the dangers of bogus treatments. It’s good when doctors educate people about the benefits of evidence-based treatments. It’s astonishing that this is controversial.
The lessons of skepticism part 2: Some people are persuadable if they are given accurate information in a clear and thorough manner
Additionally, critics of skepticism often infantilize anyone who uses alternative medicine, scornfully treating them all as unreachable naïfs. One such podcasting oncologist said, “You can’t waste all your time arguing with people who do not accept evidence and rationality as a precondition to debate. You’re wasting your time…They’re (skeptics) are not actually successful in converting the minds of different people… You can spend your life arguing with them on all these issues, but you’re not going to achieve any traction.”
This bold claim was made without any evidence, and as tends to be the case with such claims, it is wrong. We can’t reach everyone, but some people are persuadable if they are presented with accurate information in a clear and thorough manner. Even though few Shivambu devotees would have their mind changed by an article here, some might! For example, there are multiple accounts of people deciding to vaccinate their children after learning the facts. It’s a big deal when people are steered away from quackery and towards evidence-based treatments. I am certain that Science Based Medicine has done just that. My friend Amanda said the following in response to this podcaster:
Any defense of skepticism could legitimately end here, and I imagine the long-time writers here would say stories like Amanda’s alone are worth all of their effort.
Beyond this, doctors routinely treat people who use alternative medicine and are wary of evidence-based treatments. Very few of them are as far down the rabbit-hole as Dr. Brogan, and many are open to facts. Doctors need to be able to relate to their patients and help them make wise decisions. We can do this only if we make some minimal effort to understand where they are coming from, especially during a pandemic where millions of people are refusing to be vaccinated. I am very familiar with anti-vaccine disinformation, and I’ve familiarized myself with techniques to help people who are skeptical about vaccines. Hopefully, I’ve helped my patients reject the myths they encounter online, some of them inspired by contrarian doctors I’ve discussed here before. Doctors who thought that learning about alternative medicine was beneath them are likely not as prepared to do this.
I fully agree that skeptics have to be thoughtful in how they discuss these issues. Exposing and mocking Dr. Brogan is appropriate. People need to know that doctors who actually care for patients view her as an object of ridicule. However mocking someone who is genuinely confused about alternative medicine isn’t going to help anyone. As Dr. Steve Novella wrote recently, “ordinary people who believe, use, and even spread false medical beliefs are victims. They do not deserve our condemnation, but our sympathy.” Given that some prominent doctors have spread misinformation, repeatedly minimizing the virus and hyping every vaccine side-effect, it’s not a surprise that many people are legitimately confused about whether or not the vaccine is beneficial for young people (it is). I also agree that skeptics need to be wary of amplifying pseudoscience that would have gone unnoticed otherwise. The best way to refute a lot of nonsense is to ignore it and let it quietly die on its own.
However, I wonder where people like Amanda would be if Dr. Brogan and other members of the Disinformation Dozen had their deluge of disinformation greeted with complete silence by the medical community. If ethical doctors said nothing, the countless grifters, frauds, and quacks would have the internet to themselves, completely unopposed by rational voices.
While this may not be the desired goal of skepticism’s critics, it is the obvious end point to their position. Either they don’t recognize this or don’t care about it. This too is astonishing.
The lessons of skepticism part 3: Skepticism is an allegiance to a method of inquiry rather than any answer
Ideally, a true skeptic would not start out trying to debunk Shivambu. Rather, a true skeptic would realize her preconceived notions could be wrong, even about something that seems as absurd as Shivambu. After all, stool transplants are now the standard of care for treatment-resistant C. difficile infections. As such, our true skeptic would first acknowledge her biases, and she would do her best to investigate Shivambu with an open mind. She’d examine the all the evidence, and she’d report her findings as fairly as possible. If new evidence emerges that contradicts her initial conclusions, she would acknowledge it and adjust her opinion accordingly.
Skepticism then is fundamentally about allegiance to a method of inquiry, not to any final answer. For example, my first article here argued that children should be vaccinated against COVID-19. My second article reported that incidental COVID-19 hospitalizations were common in children, and I’ve since discussed evidence that the pediatric death toll may be inflated, as well as vaccine-related myocarditis. Though the virus remains much more dangerous than the vaccine, these subsequent articles all weakened the case to vaccinate children. However that’s what the evidence showed, and so without any hesitation, I reported it as thoroughly and honestly as possible. I’ve done my best not to overstate the harms of the virus or understate flaws with the vaccine. As an all-around scold, I’ve called out people several times for overstating the harms of COVID-19 to children. I’ve written about my own errors (here and here). My loyalty is to a process, not a result
Because of this, I’ve defended people (here and here) who were not in my “tribe,” and I’ve argued with people who are in my “tribe”. I’m glad they argued back. Of course, skeptics bash quacks like Dr. Brogan, but they also disagree with each other all the time. For example, Dr. Gorski recently called out Richard Dawkins for embracing the lab leak theory based on a shoddy documentary. To the extent I have a “tribe,” it consists of people who are not afraid to thoughtfully call me out when they think I am wrong.
In contrast, pseudoscience vendors like Dr. Brogan start with the answer they want, and work backwards to find the “evidence.” She readily admits this, saying, “It’s time to decide what you believe. And make no mistake, it’s about belief, not about facts, not about science as the final objective.” Anyone and anything that threatens her desired beliefs is wished away as a conspiracy or ignored entirely. Because she defines what is true as what she wants to be true, not only will she never admit error, she essentially feels she’s incapable of being wrong to begin with. There’s literally no evidence or study that would lead Dr. Brogan to change her mind about Shivambu, vaccines, or basically anything at all. That’s a problem, but not one that is unique to Dr. Brogan.
The lessons of skepticism part 4: By understanding the flaws in alternative medicine we can identify flawed medical practices elsewhere
Having learned to evaluate the claims of alternative medicine, the skeptics’ next hope is that they can identify and avoid flawed medical practices elsewhere. Someone who can spot the defects in acupuncture studies published in prestigious medical journals will be prepared to spot defects in wide swaths of medicine. Skepticism’s methods work not just for critical appraisals of Shivambu, but also for problematic drug approvals, useless medical procedures, cancer screening, and flawed research methodologies. Skeptics discuss these topics often, despite what their uninformed critics may claim.
I think my exposure to skepticism, mainly reading Science Based Medicine, helped me be a more thoughtful doctor. For example, at the start of the pandemic, I had one demand when the medicine service asked me to run a COVID-19 team: I would not practice alternative medicine. As I wrote previously,
I also made sure that no patient on my team received any experimental treatments (such as hydroxychloroquine). Doing “nothing” is often very difficult for doctors, but is preferable to giving out unproven treatments unless patients are clearly dying or in a clinical trial.
Though they would have denied it, many doctors were practicing alternative medicine at that time. I understand why. COVID-19 was a horrifying new disease and we were desperate save lives. However, I recognized that doctors who used unproven treatments like hydroxychloroquine were no different than alternative medicine practitioners, and I wanted no part of that. Learning about the flaws in practices like Shivambu helped me make better choices for my patients – at least I hope so – and I suspect fewer doctors would have used dubious treatments early in the pandemic if they had a better understanding of alternative medicine.
The lessons of skepticism part 5: There is no such thing as a harmless alternative treatment
Another core goal of skepticism is to show that while any individual practice in alternative medicine may be harmless, the overall practice of science-denial is not harmless at all. There is a significant price to be paid when a large percentage of the population lacks the basic scientific knowledge and critical thinking skills needed to evaluate the claims of alternative medicine. The same distorted thinking that allows people to believe in Shivambu often leads them to reject evidence-based treatments such as vaccines, something called crank magnetism. The anti-vaccine movement cannot be disentangled from the seemingly benign treatments in alternative medicine. Believing in “harmless” things is harmless until it leads tens of millions of people to reject vaccines and embrace quack therapies in the middle of a raging pandemic.
As such, skeptics would argue that there’s really no such thing as a “harmless” alternative treatment. Anything that weakens people’s ability to think critically about medicine and health is dangerous. Indeed, wellness influencers who promoted crystal-healing and “clean-eating” in 2019 are comparing vaccines to the Holocaust today, an unholy union of noxious beliefs termed conspirituality.
Skeptics are not surprised by any of this, and many sounded the alarm prior to the pandemic. In a conversation from 2019, Dr. Mark Hoofnagle said, “Science communicators have been warning about the dangers of conspiracy theorists for decades now. AGW, antivax, evolution denial – all the same. Here we are. This is what it looks like when conspiracy theorists take over. They are the worst people.” I agreed and added that “the biggest danger to my kids isn’t the measles, but the thought process that leads people to reject the vaccine in the first place.” Little did we know.
In contrast, our pollyannaish podcasting oncologist actually thought not long ago that children’s “risk of covid will decline precipitously” after “all adults are vaccinated“. Only someone with a complete ignorance of the depths of anti-vaccine sentiment in the US would believe that adult vaccination rates would be high enough to obviate the need to vaccinate children. I was under no such illusions, even before the Delta (let alone omicron) variant arrived.
Rachel Moran, who studies medical misinformation, explained the scope of the problem well when she said,
Medical misinformation doesn’t just result in people making bad personal and community health choices, but it also divides communities and families, leaving an emotional toll. Misinformation narratives have real sticking power and impact people’s ability to make safe health choices.
She’s absolutely right. While there are complicated and varied reasons behind vaccine-refusal, at the end of the day, 1,000 Americans are dying daily because they declined a safe and effective vaccine against a deadly virus. Science denial is a leading cause of death right now, and children have died because their parents declined to have them vaccinated against COVID-19. Skepticism’s detractors survey this carnage and are mostly perturbed that someone might use an hour of their free time to debunk Shivambu.
Prior to the pandemic, several doctors mocked skeptics who discussed topics such as Shivambu. They felt it was a waste of time and easy to do, likening it to dunking on a 7-foot hoop. They felt that warning about bogus cancer cures took time away from more urgent, challenging tasks, such as dunking on bad studies on coffee, Dr. Oz and skeptics, I guess.
I fully agree with skepticism’s critics about one thing however; in one sense, it’s extremely easy to debunk Shivambu. A doctor whose priority is to flaunt his research prowess would not tackle this topic. It takes no special abilities to quickly discover the unsurprising fact that there is no evidence drinking one’s own urine cures cancer. I am certain my teenagers could do this.
However, I do not think my children would be able to communicate about Shivambu in a thoughtful manner to a newly diagnosed cancer patient who had stumbled on to Dr. Brogan’s interview. That’s an entirely different skill set. Moreover, my children would not be able to think deeply about the broader implications of Shivambu. They would not be curious as to why some smart people believe drinking urine cures cancer. They would not be able to place Shivambu into the wider context of the rampant science-denial and institutional mistrust in the world today. They would not be able to link Shivambu to 1,000 people dying everyday of a vaccine-preventable disease, to violence in the anti-vaccine movement, or to an anti-vaccine doctor being arrested for her role in the January 6th insurrection. They wouldn’t be able to see it as a symptom of the dangerous world of alternative facts that has enveloped much our culture, including doctors I’ve discussed before. They wouldn’t grasp that a fantasy world where drinking urine cures cancer is also a world where COVID-19 is a hoax that is cured by ivermectin and there’s no reason to get vaccinated.
It’s good that there are doctors who can see these connections and who care enough to push back against the misinformation that creates such delusions.
The lessons of skepticism part 6: We are all in danger of becoming 7-foot hoops
The skeptic’s final goal is to examine the cognitive flaws and distorted thinking that leads smart people to believe that drinking urine can cure cancer. Dr. Brogan went to MIT where she studied systems neuroscience. She then attended Cornell medical school and graduated NYU residency with me. She’s many things, but she’s not stupid, even though she’s come to believe really stupid things. The skeptic’s aim is therefore not to just understand what is wrong with Shivambu, but also to understand what is wrong with Dr. Brogan. And since we all have cognitive biases and blind spots, we need to understand what is wrong with Dr. Brogan to avoid becoming like Dr. Brogan.
Were skepticism’s detractors to read an article debunking Shivambu, they’d merely see it as an article debunking Shivambu and nothing else. I think these concrete-minded critics missed the larger points skeptics were trying to make, and as a result, they missed out on what skepticism has to offer. Put another way, people who think they’re above learning about 7-foot hoops are in danger of becoming 7-foot hoops.
However, skepticism teaches us that we are all in danger of becoming 7-foot hoops. Unfortunately, it’s easy to find examples of smart people who’ve suffered this fate. Just think about Dr. Oz or Dr. Joseph Ladapo. Pseudoscientific treatments are routinely offered by highly-educated doctors at prestigious medical centers, a sad phenomenon dubbed “quackademic medicine“. Doctors who were widely-admired vaccine advocates prior to the pandemic have since repeatedly minimized the virus and mocked those who, rightly, warned about its dangers, calling them prophets of “doom and fear”. Even brilliant Nobel Prize laureates have claimed that vitamin C was a magic cure-all, that HIV doesn’t cause AIDS, that Black people are less intelligent, that COVID-19 vaccines create new variants, and that COVID-19 was much less dangerous than it was made out to be. Like I said, really smart people can believe really stupid (and horrible) things.
Perhaps because they know they are smart, intelligent people may erroneously feel they are less vulnerable to biased thinking than everyone else. This is itself a bias, one that skeptics are keenly aware of. Almost all skeptics enjoy optical illusions and magic. They love learning how easily we can be fooled, and how we are often completely unaware of it. Dr. Novella called awareness of these vulnerabilities neuropsychological humility and defined it thusly:
Being a functional skeptic requires knowledge of all the various ways in which we deceive ourselves, the limits and flaws in human perception and memory, the inherent biases and fallacies in cognition, and the methods that can help mitigate all these flaws and biases.
Given that no one is immune from biased thinking, we all need to be open to the possibility that we could be wrong. Psychologist Adam Grant shared two wise thoughts on this theme recently.
The hallmark of an open mind is not letting your ideas become your identity. If you define yourself by your opinions, questioning them is a threat to your integrity. If you see yourself as a curious person or a lifelong learner, changing your mind is a moment of growth
It takes humility to consider information that contradicts your opinions. You’re willing to concede that you might be wrong. It takes curiosity to actively seek evidence that challenges your views. You’re eager to find out if you might be wrong.
He’s right. Just because I sanctimoniously boasted that “My loyalty is to a process, not a result” doesn’t mean I’m actually good at that process. I know I’ve fallen short at times.
We all need to be humble and listen to our well-meaning critics. Oftentimes only they can show us our errors. It’s a serious mistake to insulate yourself in an echo chamber of sycophants, dreaming you are persecuted whenever you are contradicted. Oddly enough, Dr. Brogan was on to something – it is very easy to let your beliefs replace facts and science, especially when these beliefs lead to financial rewards, TV appearances, and social media adulation. We all need to be continuously on guard against this.
Richard Feynman famously said, “The first principle is that you must not fool yourself—and you are the easiest person to fool.” This is the most important lesson skepticism has to offer.
Contrarian doctors and Dr. Brogan part 1
Sadly, several contrarian doctors haven’t grasped skepticism’s lessons, and so they seem to have fallen into the same cognitive traps as Dr. Brogan. They have an almost petulant need to be different, no matter the evidence. One Twitter commentator wryly called this “obligate contrarianism: The need to be contradictory no matter what the mainstream stance is”.
They are hostile to criticism, claiming that anyone who challenges them is trying to “quiet” them. They conflate criticisms of their ideas with personal attacks, and so instead of engaging with those who disagree with them, they ask to be left alone. They brush off all who disagree with them with unsubstantiated accusations of “tribalism” and “groupthink.” They rarely, if ever, admit error, and most tolerate only fawning lickspittles on social media. They mock people who fear or warn of the virus, implying that life is passing them by or that they are addicted to doom and being on TV. Many have repeatedly declared the pandemic over, though they continue to talk about it constantly. They have a shared sense of grievance, believing they are the real victims of this pandemic because YouTube removed some of their videos – and boy do some of them make a lot of videos. At the risk of sounding like a conspiracy theorist myself, several have connections to shady right-wing think tanks.
Like Dr. Brogan, several contrarians started the pandemic with the belief that COVID-19’s harms were greatly exaggerated (here and here), and they’ve spent the past 20 months floundering to justify this initial, flawed premise. One contrarian has peddled identical conspiracy theories as Dr. Brogan, claiming that death certificates have been unreliable this pandemic (in direct contradiction to the evidence), and the COVID-19 death toll is inflated. He wants you to believe many people died with COVID-19, not from COVID-19 and that healthcare workers have perverse financial incentives to put COVID-19 on death certificates. Even worse than this, he repeatedly claimed, again in direct contradiction to the evidence, that doctors killed patients by intubating them too quickly. All this to avoid saying, “I underestimated COVID-19”.
He is not alone. Another noted early in the summer that “cases are FALLING PRECIPITOUSLY” and that for children, “COVID and MIS-C is plummeting with adult vaccination“. He used these numbers to justify his belief that children should not be vaccinated against COVID-19. When cases were at their nadir, he wrote a lengthy piece criticizing the CDC for using outdated numbers when modeling the benefits of the vaccine for adolescents. He called this a “noble lie”. However, when pediatric cases and hospitalizations spiked to their highest levels soon thereafter, he then said that hospitals “admit less sick patients” to for financial gain. Rather than admit that he too had underestimated the virus, he claimed the number of patients hospitalized with COVID-19 is an “inaccurate” metric and is instead merely a “fear monger tactic”. Beyond this, he fantasized that public health measures to contain a virus could one day lead to totalitarianism.
Dr. Brogan wouldn’t object to any of this. She believes the entire pandemic is a fear monger tactic for profit, and that totalitarianism has already arrived, saying COVID-19 was planned for the “entrapment of citizens worldwide.” Their combined message is: Don’t believe news reports of overwhelmed hospitals. Don’t believe what doctors in the trenches are saying. They are out to deceive and scare you for profit. Only I will tell you the truth. Purchase a subscription to my newsletter/online course or pay my speaking fees to gain access to my unique insights.
Contrarian doctors and Dr. Brogan part 2
Nowhere is the overlap between contrarian doctors and Dr. Brogan more complete, however, than their shared stance on the risk of COVID-19 to children and whether or not they should be protected against it. There’s no question these contrarian doctors have more in common with her than with the American Academy of Pediatrics, the CDC, and almost every practicing pediatrician in this regard. That says a lot considering Dr. Brogan is a germ-theory denier who does not believe HIV causes AIDS, that SARS-CoV-2 causes COVID-19, or that we are experiencing “a real medical pandemic“.
However, these contrarians and Dr. Brogan all agree that COVID-19 poses minimal-to-no threat to young people and measures to protect them against the virus will likely do more harm than good. They felt this spring that children should not get the COVID-19 vaccine. Once adolescents started getting vaccinated, they felt that “the US should suspend all vaccination in children under 18“. They agreed that “the downsides of masking young students are real“. They agreed that “Children should return to their normal lives this summer and in the upcoming school year, without masks and regardless of their vaccination status“. It’s no surprise that their ideas get favorably shared (here and here) by anti-vaccine cranks like RFK Jr., who posted an article by some of these doctors arguing against the Emergency Use Authorization of the COVID-19 vaccine for most children. Thank goodness no one with real power took that idea seriously.
Highly credentialed doctors from Stanford and Harvard penned Broganesque articles such as “The Ill-Advised Push to Vaccinate the Young“. This article opened by saying:
The idea that everyone must be vaccinated against COVID-19 is as misguided as the anti-vax idea that no one should. The former is more dangerous for public health.
That’s right. These doctors believe the idea of nobody getting vaccinated is better for public health than the idea of everybody getting vaccinated. In the middle of the worst pandemic since 1918, these doctors feel that vaccine-advocates are a greater menace to public health than anti-vaxxers. I’m curious to know what they think the folks at Voices for Vaccines might do that would lead to over 1,000 Americans dying every day for months at a time. That’s the toll of the anti-vaccine movement at present. What great evil do they think Karen Ernst is capable of? She’s always seemed nice to me.
This article also claimed that mandating the vaccine for young people “threatens not only public health but also the future of science“. Elsewhere, one of the authors even spread the absurd myth that vaccine mandates will actually spread the virus. These histrionic, preposterous statements seem plagiarized from the most deranged anti-vaccine websites. Not surprisingly, Dr. Brogan also opposes ‘public health measures imposed in violation of bodily sovereignty (for example, mandated vaccination in exchange for “privileges” which may actually be inalienable rights)’.
Contrarian doctors and Dr. Brogan part 3
By the time this article was published, hundreds of American children had already died of COVID-19 and many thousands more had been hospitalized. Though I agree that the virus poses an “extremely low” mortality risk for children, millions more children contracted the virus after this article was published, and so hundreds more children died, including some without underlying conditions. Young adults suffered even more. According to the CDC’s Covid Data Tracker, just under 6,000 Americans younger than 29 years have died of COVID-19 thus far, including over 1,000 children. All of these young people died from the virus, not the vaccine.
Tens of thousands more children and young adults have been hospitalized. Though they thankfully survived, many were very sick, needing ICU-level care and mechanical ventilation. Some pediatric COVID-19 survivors suffered strokes or encephalitis. Others needed lung transplants or amputations. Their lives will never be the same. 5,973 children have had MIS-C thus far, though this may be a substantial undercount. In one study, 80% of children with MIS-C went to the ICU and 20% needed mechanical ventilation. 52 children have died of MIS-C. Nearly all these tragedies occurred in unvaccinated people, many after the vaccine was widely available. Thankfully, these tragedies remain very rare. Almost all young people with COVID-19 will be fine.
Fortunately, the vaccine has proven highly effective in keeping adolescents alive and out of the hospital. Hospitals were deluged with children suffering from COVID-19, not side effects from the vaccine. One study found that “Hospitalization rates were 10 times higher among unvaccinated than among fully vaccinated adolescents”. Another study of adolescents showed the efficacy of the vaccine in this age group. It found:
Among 179 COVID-19 case-patients, six (3%) were vaccinated and 173 (97%) were unvaccinated. Overall, 77 (43%) case-patients were admitted to an intensive care unit, and 29 (16%) critically ill case-patients received life support during hospitalization, including invasive mechanical ventilation, vasoactive infusions, or extracorporeal membrane oxygenation; two of these 29 critically ill patients (7%) died. All 77 case-patients admitted to the intensive care unit, all 29 critically ill case-patients, and both deaths occurred among unvaccinated case-patients.
There’s good news about vaccine-safety as well. Over 7 million doses have been given to children ages 5-11 years, including over 2 million second-doses, and there have been only 8 verified reports of myocarditis so far with 6 more cases under investigation.
In addition to this, evidence continues to accumulate that almost all young people with vaccine-myocarditis have a “mild” course and recover well. Dr. Dongngan T. Truong, who authored a large case series on the topic said “most cases of suspected COVID-19 vaccine-related myocarditis in people younger than 21 are mild and resolve quickly”, though he acknowledged “we are awaiting further studies to better understand the long-term outcomes of patients who have had COVID-19 vaccination-related myocarditis”. The Pfizer vaccine seems safer than the Moderna vaccine for young men, as does using a longer interval between the two vaccine doses. With this knowledge, it’s possible to take a low-risk event and make it even lower. I am aware of literally just one death of a young person (a 22-year-old) in the entire world from the mRNA vaccines. There’s no question that the vaccine is safer than the virus for young people.
Neither Dr. Brogan nor these contrarian doctors want you to know or even believe any of this. Since they only report information that bolsters their preconceived notions, they will never mention most of these facts. If you encounter accurate information elsewhere about pediatric hospitals being overwhelmed with COVID-19 cases, they’ll claim they the numbers are a “fear monger tactic”. If you believe the numbers, they’ll claim that COVID-19 can’t be that bad for kids because it’s worse elderly people (true, but irrelevant), that more children die of suicide (true, but irrelevant), that the flu is worse (false), and that it’s only children with leukemia who are dying (false and repellent). Others have “wondered” whether incompetent pediatricians were misdiagnosing RSV as COVID-19 (they weren’t). Several doctors simply spread wrong blatantly wrong numbers, always minimizing COVID-19’s impact on children, of course. I’ve even seen some contrarians claim that hundreds of dead children from COVID-19 can’t be that bad, because after all, there are 74 million kids in the USA.
What is so different about COVID-19 that makes it so easy for these doctors to brush off the deaths of children? Though almost all children with COVID-19 do well, enough of them have contracted the virus to make it a leading cause of pediatric deaths this year. These tragedies can now be prevented with a vaccine. Imagine if we could prevent suicides, accidents, and drownings with a vaccine. We’d do it, right? However, beyond magically isolating themselves from older people, these doctors don’t want most children to do anything to prevent COVID-19 – no masks, no vaccines, no testing, nothing. Meanwhile, the American Academy of Pediatrics keeps adding 2-3 deaths per day to its grim tally.
Imagine the outrage there’d be if a doctor tried to diminish the tragedies of pediatric cancer the way pediatric COVID-19 deaths are routinely minimized. Imagine if after the next school shooting some doctor pointed out that 99.999% of children won’t die this way, so we shouldn’t do anything to try to prevent these tragedies. Maybe more people need to read about the individual children who have died of COVID-19 (Skylar Herbert, Tagan White, Kimora Lynum, Wyatt Gibson, Zyrin Foots). These were real lives, cut tragically short, not just numbers in a CDC dataset. Their deaths shouldn’t be diminished just because their grandparents were statistically more likely to die. Again, what makes 1,000 pediatric COVID-19 deaths so uniquely tolerable, especially considering the only unique thing about them is they can be now prevented with a vaccine?
Though they claim to be “pro-vaccine“, it seems these contrarians will do anything to avoid acknowledging an obvious truth: COVID-19 can be a devastating, even fatal, disease for a very small percentage of young people, and the vaccine can prevent these tragic outcomes. In contrast, they are extremely eager to discuss vaccine side effects. One doctor even repeatedly mentioned post-vaccination fevers as a reason not to vaccinate children. Just a few short months ago, doctors were arguing in a still unpublished pre-print that the vaccine could cause more harm than the virus for adolescent boys. It’s likely that the number of times they’ve collectively mentioned vaccine-related myocarditis (here, here, here, here, here, here, here, here, here) exceeds the number of children currently hospitalized with the condition. However, since vaccine-related myocarditis almost always has a very favorable clinical course, they are generally only willing to discuss the frequency at which this condition occurs. One doctor even used the death of that single 22-year-old to argue the vaccine could kill as many children as the virus. Studies showing the vaccine’s significant benefits for adolescents are, of course, completely unmentionable.
Because their loyalty is to a result, not a process, these doctors have consistently refused to enumerate the harms of the virus and the benefits of the vaccine. As a result, their writing on COVID-19 and children is uniformly uninformative and devoid of nuance, especially compared to the work of doctors familiar with skepticism and critical thinking. After the Delta variant hospitalized many thousands of children and killed hundreds of them, some contrarians belatedly altered their blanket opposition to vaccinating children, though they did so without acknowledging their previous stance and what role this vocal opposition might have had in creating significant vaccine-hesitancy in parents. However, as with Dr. Brogan, there’s literally no evidence or study that would lead these doctors to conclude it was a mistake to have left tens of millions of young people vulnerable to COVID-19.
After all, if the deaths of thousands of young people, including 1,000 children, doesn’t give them pause, what possibly could?
Contrarian doctors and Dr. Brogan part 4
Though Dr. Brogan and these contrarians have little credibility in the medical community, like alternative treatments, they are not harmless. They have stellar credentials. They all have large audiences. People listen to what they say. They speak without a trace of doubt. They can easily speak in scientific jargon, misrepresenting information to make dangerous ideas sound reasonable to those who are ignorant of the facts.
Their conviction that it was preferable to leave young people vulnerable to COVID-19 convinced politicians like Florida Governor Ron DeSantis to let the virus rip through young people in his state. Dr. Brogan has had more impact at the local level in that state as she influences anti-vaccine schools in Miami. Thanks in part to their combined efforts, children’s hospitals were overwhelmed at the end of this summer. Some children died, including healthy teens who were eligible to be vaccinated. Their ideology also seems to have influenced the UK’s response to vaccinating children, with disastrous consequences I discussed previously.
The doctors are all generally sheltered from the consequences of their words. Dr. Brogan has never cared for a patient hospitalized with COVID-19, and neither have most contrarians. They’ll likely never see a young person gasping for air, regretting their decision not to get vaccinated. The pandemic exists entirely on a computer screen for most of them. Despite this, they remain defiantly proud of their efforts, calling Florida a “success“. It’s frightening to contemplate, but their influence may grow significantly if the political winds blow their way.
I was friendly with Dr. Brogan during her time at NYU. It wasn’t until after she graduated that she decided to reject “facts” and “science as the final objective”. I similarly admired some of these contrarian doctors before the pandemic. Like Dr. Brogan, they aren’t stupid. It’s certainly sad they’ve come to share many of her beliefs on COVID-19 and children. COVID-19 will fade into the background one day, and these contrarians will be remembered only for their poor predictions, vaccine fear mongering, and relentless minimization of virus that was benign for most young people, but because it was encouraged to spread so widely, managed to kill thousands of them and injure many thousands more.
Who knows? Maybe they could have avoided this sad fate if only they hadn’t thought it was beneath them to debunk Shivambu.

Revisiting the question of debating science deniers
Wealthy tech bro turned antivax crank Steve Kirsch attacked Paul Offit for refusing to debate antivaxxers, while Dr. Mike was “surrounded” by MAHA stans. These recent events led me to revisit the question: Is it ever a good strategy to publicly debate cranks?
David Gorski on January 5, 2026
Link

As I approached the task of writing my first post of 2026, I thought I’d back up a bit. Rather than dive into a detailed deconstruction of, for instance, a prominent cancer researcher and cancer center director‘s very recent doubling down on arguing that the scientifically unsupported claim that COVID-19 vaccines cause “turbo cancers” is scientifically plausible (which I might have to address another time to explain again why it’s not), I thought I’d revisit a question that I’ve written about on several occasions over the last 20 years, both here and at my not-so-secret other blog, namely the question of whether it is ever worthwhile, effective, or a good idea for scientists, physicians, and science communicators to agree to debate science denying cranks. Longtime readers of this blog will know that I have long tended—with caveats—to come down on the “No!” side of this debate. Specifically, I’ve long viewed “live public debates” of this nature less as scientific debates and more as spectacles craved by science deniers to rally their supporters by providing their viewpoints with seeming legitimacy. Science deniers accomplish this by enticing real scientists to appear on the same stage with them, where they can Gish gallop and empty deceptive rhetoric to produce the illusion that there is a real scientific debate about, for instance, the safety and efficacy of vaccines. Scientists unprepared to confront Gish gallops and rhetoric over substance can stumble, sometimes badly, which is another reason why such debates tend to be a triumph of influencer “content” over substance. Indeed, nearly 13 years ago, I coined a corollary to crank magnetism that I dubbed, “All Truth Comes From Public Debate.”
I still believe that, in general, it’s usually a bad idea for scientists to agree to these sorts of debates (albeit perhaps less vehemently than before, and recognizing exceptions). However, I now feel that a couple of recent events make it worthwhile for me to revisit this question. The first was a post by our old “friend,” tech bro turned antivax influencer and activist Steve Kirsch, who attacked pediatrician and vaccine scientist Dr. Paul Offit for having declined a challenge to engage in a public debate about vaccines made by Kirsch, longtime antivax activist now sadly in charge of the Department of Health and Human Services, Robert F. Kennedy Jr., and his collaborating lawyer Aaron Siri, who recently wrote a book calling vaccines a “religion.” In his post, Kirsch, had offered Dr. Offit $1 million and, as tends to be his wont, tried to portray himself as the true champion of science and Dr. Offit’s refusal as being unscientific.
The second incident was the appearance of Mikhail “Dr. Mike” Varshavski, DO, a family medicine doctor who’s become a well-known YouTuber and podcaster, whose The Checkup with Dr. Mike boasts over 14 million YouTube subscribers and who is known for debunking health misinformation, on Surrounded. Surrounded, for those not familiar with it, is a very popular web series/YouTube show produced by the political YouTube channel Jubilee Media. Its gimmick is to pit one person against up to 25 “debate” opponents who are seated “surrounding” the main debater, who sits at a table in the center. Opponents take turns stepping into the center to sit in a chair across from the featured debater and “debate” him or her for a set time (complete with timer to say when time’s up). Most of the topics debated this way on Surrounded tend to be political, and recent episodes that I’ve briefly perused feature unpromising titles like Piers Morgan vs. 20 “Woke” Liberals, 1 Atheist vs. 25 Christians, 1 Journalist vs. 20 Conspiracy Theorists, 1 Liberal Teen vs. 20 Trump Supporters, and, a year ago, 1 Conservative vs. 25 Liberal College Students (Feat. Charlie Kirk), which appears to be the first episode ever of this series. However, occasionally the show looks at health-related topics, such as a previous appearance by Dr. Mike several months ago (Dr. Mike vs. 20 Anti-Vaxxers) and, less impressively, Bryan Johnson vs. 20 Skeptics. (Bryan Johnson, in case you don’t remember, is the wealthy biohacker who has been in the news for spending $2 million a year to reverse the clock using quackery such as transfusions with his son’s youthful blood, full plasma exchange, a boatload of unproven supplements and treatments, rapamycin for “anti-aging” and claims that he will achieve immortality by 2039.
Dr. Mike appeared on an episode of Surrounded in December entitled 1 Doctor vs 20 RFK Jr. Supporters, where, as the title says, he confronted 20 “Make America Healthy Again” (MAHA) supporters of our current HHS Secretary. I must admit that, before this episode, I had never been able to sit through an entire episode of Surrounded, and, I must also admit, that even I found it quite difficult to sit through all of Dr. Mike’s appearance. (The episodes tend to be 90 to 120 minutes long.) However, I have to concede that Dr. Mike acquitted himself extremely well, far better than I would have expected anyone to be able to do in such a format. As a result, I started to question, if just a bit, whether my stance regarding these sorts of “debates” has been too dogmatic, even as I realized that Surrounded is about as pure a distillation of “debate me, bro” culture as one will ever come across on social media. After all, if you peruse the list of Surrounded episodes, you will find that the producers appear not to care all that much about truth; they appear far more interested in generating spectacle, arguments, and sparks, the better to attract clicks and eyeballs.
On the other hand, Steve Kirsch’s clumsy, heavy-handed attempt to brand his desire for “live public debates” with scientists like Dr. Offit and Dr. N. Adam Brown, who had penned an article criticizing RFK Jr. for MedPage today (entitled provocatively, I Was Wrong About RFK Jr.), as “science-based” and Dr. Offit’s assertion that science is best debated in the peer-reviewed literature as unscientific reminded me what the real purpose of cranks wanting “live public debates” always is. I like to refer, for instance, to Neil deGrasse Tyson’s disastrous “debate” with antivax activist Del Bigtree as a counterexample and cautionary tale regarding such “live public debates.”
With this as the backdrop, I looked at the evidence regarding the effect of “live public debates” and whether there really is such a thing as the backfire effect, in which such debates supposedly reinforce, rather than weaken, the beliefs of science deniers. Unsurprisingly, Steve Kirsch, being Steve Kirsch, cherry picked evidence and commentary and, risibly, used AI to generate arguments for his position, as though AI were any authority. Even so, I found my position regarding these debates softening a bit, not because I suddenly think they’re a good idea, but rather because I don’t think we as science communicators can avoid them entirely.
Let’s start with Kirsch.
Quoth Steve “debate me, bro!” Kirsch: I’m the real champion of science. (He’s not.)
Three weeks ago, Dr. Offit posted an article on his Substack, with accompanying video, entitled Debating Science: No Thanks, where he added, “Anti-vaccine activists RFK Jr., Aaron Siri, and Steve Kirsch have asked that I debate them on vaccine safety. One of them offered me $1 million to do it. Here’s why I declined.” In his post, Dr. Offit asserted:
Several years ago, people representing RFK Jr., who at the time was head of the anti-vaccine group, Children’s Health Defense, offered me $50,000 to debate him about the safety of vaccines. In the past week, Aaron Siri, a personal-injury lawyer whose firm represents an anti-vaccine group called Informed Consent Action Network, also asked me to debate him. As did Steve Kirsch, an entrepreneur and co-inventor of the optical mouse, who offered me $1 million if I agreed. I turned them all down because it doesn’t matter what they say during the debate. And it doesn’t matter what I say. The only thing that matters is the strength and reproducibility of the evidence.
It must be said right here that this is almost exactly what I tend to say regarding such “debates.” One also must remember that Kirsch is a hard core conspiracy theorist about vaccines. Early in the pandemic he founded the COVID-19 Early Treatment Fund (CETF) in order to fund research into off-label treatments for COVID-19 using existing drugs already having FDA approval for other diseases. Unfortunately, his reactions to the results of his project, the good and the bad, were a major foreshadowing for the antivax heel turn that he would soon take. Refusing to believe the results of a study funded by CETF that found that hydroxychloroquine had no value treating COVID-19, he turned on the investigators, accusing them of poor study design and statistical errors. By mid-2021 Kirsch was promoting ivermectin as a miracle cure for COVID-19, that COVID-19 vaccines impair female fertility (they don’t), and that the mRNA vaccines were killing more people than they were saving—150,000 killed by the vaccines, according to him! By May 2022 he had upped that number to claim that mRNA vaccines had killed over a half a million people while only saving ~25,000.
By 2022, Kirsch had developed a penchant for challenging critics of COVID-19 quackery and antivax misinformation to live public debates, a penchant that led him to annoy the crap out of legitimate scientists by issuing fatuous “challenges” conveyed to them by email and publicly on his Substack. (I’ve been at the receiving end of a few.) Equally predictably, Kirsch’s anti-COVID-19 vaccine variety of antivaccinationism soon expanded to encompass “old school” antivax conspiracy theories, such as the CDC whistleblower conspiracy theory featured in the 2016 antivaccine propaganda “documentary” VAXXED (even more recently resurrected by Robert F. Kennedy, Jr.) and, of course, the long debunked claim that childhood vaccines cause autism, all while using legal bullying and doxxing to silence critics. Indeed, his MO has been to offer scientists like Dr. Offit a million dollars (or to donate a million dollars to the charity of their choice) in return for “debating” him or someone chosen by him or to bet the chosen debater a large sum of money over the results of the debate. Unsurprisingly, he’s been accused of welching on at least one of his bets, leading Kirsch to dox him and threaten to sue him. One cannot forget this background when considering anything Kirsch says about debating scientists. Context matters.
But back to Dr. Offit. He made what I consider to be a reasonable response when faced with a “challenge” to a “debate” about vaccines from the likes of Aaron Siri, Steve Kirsch, and RFK Jr. Pointing out the distortions and misrepresentations of vaccine safety data by Aaron Siri and RFK Jr., he wrote:
Enter Steve Kirsch, who has said that Pfizer’s Covid-19 vaccine has “killed more people than it saved.” Great. Prove it. Do the kinds of epidemiological studies that were done for Guillain-Barré syndrome, intussusception, narcolepsy, and clotting. Do the hard work that’s required to prove that a vaccine safety problem is real and stop whining about Big Pharma and government conspiracies. Serious vaccine safety problems will be published if the data are rigorously collected, analyzed, controlled for confounding variables, subjected to peer review, and reproduced by other investigators. And they won’t be published if the data are weak. Vaccine safety is a scientific question that should be solved in a scientific venue, not on a debate stage.
Indeed. And it has been. However, that’s not what people like Kirsch, Siri, and Kennedy are about, and Dr. Offit knows it.
Which brings me to Kirsch’s response to Dr. Offit’s rejection, The peer-review literature says science should not run from public challenges; here’s a framework for fair discussions, in which he attempts to frame Dr. Offit’s refusal as being “antiscience” (as in “I know you are, but what am I?): “None of the people who claim RFK Jr is doing a bad job will consent to public challenge. That’s anti-science. Here’s a framework for fair public discussions on the most important issues of our time.”
He begins his “executive summary”:
None of the “experts” who think:
    • RFK is causing harm
    • that ACIP panel members are anti-science
    • that the COVID vaccines have saved > killed
    • that vaccines don’t cause autism or chronic disease
    • will come to the public discussion table to talk about it. That makes the problem worse, not better.
He then claims that he will “cover”:
     1.  the evidence directly from the scientific literature that refusing debate challenges is anti-science
     2.  how a fair public discussion should be structured
     3.  how the peer-review system relied on by Paul Offit and others should be reformed (one of the most important articles I’ve written to date)
     4.  my email to Dr. Brown pointing out that RFK Jr isn’t the problem; it’s that people like him refuse to engage in public discourse with qualified peers who have different views
I’ll take these out of order. Also, I’m not going to discuss #4 much. You can read Kirsch’s post, which contains a screenshot of the email, and see for yourself that the email is an attempt to shift the blame from people (like him) promoting medical misinformation to those refusing to engage in direct one-on-one live public debates with people promoting medical misinformation. Later in the post, Kirsch invokes AlterAI to deconstruct Dr. Offit’s Substack post, which never fails to amuse me whenever he does this. Seriously, dude, make your own arguments, rather than lazily prompting hallucination prone LLM-AI models to do your work for you.
As for #2, I might have to do a full post on his “proposals” to “reform” peer review at some point. However, Kirsch starts out with rants about peer review as “gatekeeping” (as though that were a bad thing), moaning that “Mainstream literature does not contain papers explicitly concluding that ‘vaccines killed more than they saved’ — because such framing will not pass peer review gatekeeping.” (I would argue in this case that gatekeeping is a good thing, because every such paper that I’ve read and deconstructed has been some of the most hellaciously bad science I’ve ever seen. Despite its shortcomings, review can work when it comes to truly bad science.) Kirsch then moves on to mixing some somewhat reasonable points (decoupling scientific publishing from corporate funding through advertising, for instance; I doubt very many scientists would disagree with the contention that big publishers might distort what is being published) with typical tech bro nonsense, like using blockchain to trace and track peer review and granting “equal access” to government datasets to science-denying cranks. (True, he calls them “independent researchers,” but I know what he really means.) Where we differ with Kirsch is that we don’t assume that this profit motive inevitably leads to the “suppression” of any data or studies that cast doubt on the safety and efficacy of vaccines, as Kirsch does. I’ve even said that a double-blind peer review process might be desirable, although I doubt that it will be always possible given that peer reviewers who are sufficiently expert in their fields to be asked to peer review a manuscript will very often be able to guess which lab or group of investigators produced a given study.
Overall, Kirsch’s criticisms of peer review and proposals for “reform” obviously derive from a science-denying antivax perspective full of typical narratives of big pharma “suppressing The Truth.” These include narratives about conventional scientists like Dr. Offit supposedly being closed-minded “gatekeepers” who reflexively reject brave mavericks like him who “question” the dominant narrative on vaccines, coupled with “prove me wrong” boasting about antivax misuse of various datasets and statistical malfeasance by antivax “scientists” to torture data into confessing what they want to hear.
More interesting is Kirsch’s statements on what a “fair” public debate. Again, as always, he mixes the reasonable with the not-so-reasonable. For instance, it is correct that a clear, narrow question is best. What he doesn’t mention is that the downside of that is that, if such a restriction is adhered to, not very much will be concluded, as the conclusion will be narrow. Of course, his example of such a “narrow” question (“Do current data justify the claim that mRNA COVID-19 vaccines are safe and effective in adults under 40?”) is not narrow at all, when you start addressing the topic in depth! He also gives the game away in his language, such as how he describes the two sides of the “panels” that will debate, describing the science-defending side as “institutional orthodoxy” and the crank side as “independent dissent.” I saw right through that framing.
Kirsch also notes:
Importantly, every participant must disclose financial and institutional conflicts of interest — grants, patents, royalties, or advisory roles. Transparency about motive is a precondition for trust.
I’m game for this, but only if everyone on the antivax side also discloses how much they make selling supplements, publishing Substacks, doing social media (such as YouTube, TikTok, and Instagram), serving as expert witnesses, and the like.
But what about Kirsch’s evidence?
Now, let’s get to the evidence that Kirsch marshals to claim that refusing such debates is “unscientific”:
See:
Public debate is good for science which was an editorial in Science, of all places encouraging debate.
Here is the actual science behind the recommendation to debate: 
Effective strategies for rebutting science denialism in public discussions. 

That paper has been cited over 204 times in the scientific literature.
So science says you don’t run from debate challenges.
The first article was written by H. Holden Thorp, the editor of Science, in early 2021, just as mRNA-based COVID-19 vaccines were starting to be distributed. It is an opinion piece. From my vantage point in 2021 after years of dealing with science deniers (and even more so since), Thorp is rather naive:
The days of going to a Gordon Conference or Asilomar and having a confidential debate about scientific issues are gone, and that’s for the best because those gatherings were not diverse enough and excluded a lot of important voices. These days, the public can access debates about science regardless of where they take place, so the medium isn’t so important anymore. What matters is getting to the right place in terms of the science—deciding what the question should be, the appropriate way of answering it, and the correct interpretation of the data. For many scientists, public debate is a new frontier and it may feel like the Wild West (it may well be). But rather than avoiding such conversations, let the debates be transparent and vigorous, wherever they are held. If we want the public to understand that science is an honorably self-correcting process, let’s do away once and for all with the idea that science is a fixed set of facts in a textbook. Instead, let everyone see the noisy, messy deliberations that advance science and lead to decisions that benefit us all.
I’m all for letting people see the messy deliberations that advance science, but, again, that’s not what antivaxxers like Kirsch are about. They’re about casting doubt on science that they don’t like, using whatever disinformation techniques they deem necessary, be it cherry picking studies, distorting data, producing bad science of their own, or attacking the scientists defending current scientific understandings. Thorp was correct that the “old days” were not diverse enough and could exclude important voices, but they had the advantage of forcing those with contrary positions to actually try to bring the data needed to cast doubt on existing science and support their positions. As we used to like to say (and I still like to say) scientific evidence talks, bullshit walks. That’s still true. It’s also important to remember that all science denial is arguably rooted in conspiracy theories, and data alone in debates of this type will not dispel conspiracy theories.
The second reference cited by Kirsch is a bit more interesting—and provocative. I can see why it appealed to him, as he is not the first antivaxxer whom I’ve seen citing this paper. In any event, it’s a 2019 paper in Nature Human Behavior by Philipp Schmid and Cornelia Betsch and, contrary to Kirsch’s characterization of it, it doesn’t quite declare that refusing debate challenges is unscientific. Rather, it’s a detailed discussion of how to effectively rebut claims of science denialists in public discussions, which include debates. As an aside, I can’t help but find it rather amusing how, by citing this paper, Kirsch seems to be implicitly admitting that he is on the side of science denial. That’s because this study is all about the effectiveness of two primary proposed techniques for refuting science denial in public discussions. Indeed, one chart is particularly useful. It’s a matrix of arguments against a hypothetical vaccine:
The five domains above, or techniques of denial, have become pretty well accepted in misinformation research, with the mnemonic FLICC (Fake experts, Logical fallacies, Impossible expectations, Cherry picking, and Conspiracy theories). Now here’s the thing. This study is not just about whether engaging in public debates with science deniers is effective; rather it’s about how to engage in public discussions, which include debates. Indeed, in the introduction, the authors explicitly state their purpose:
Beyond the question of whether to attend the discussion at all, advocates for science around the globe lack empirical advice on how to respond to a science denier in a public discussion.
Rather, the authors recognize the problems with scientists joining in such discussions and is meant to test various strategies for debunking misinformation as it happens. It’s more expansive in that it is not referring to staged debates, but to any public discussion in which a scientist is involved and encounters scientific misinformation. The authors are quite clear, noting that scientists have ignored the question of how to effectively counter the arguments for science denialism for too long and acknowledging a need for scientists to publicly debunk misinformation:
Researchers have now increased their efforts to focus on how advocates for science can inoculate individuals against misinformation before they encounter it20,21 and how misinformation can be corrected once individuals believe it22,23. A third option is to counter arguments of denial at the very moment that they reach an audience; that is, rebutting deniers in public discussions24. We will focus on this third option.
The authors also note the not inconsequential risks of this approach:
However, public discussions also entail risks for the discussants. Bad performance can, in the worst case, serve the opponent’s cause27. Moreover, backfire effects in attempts to debunk misconceptions23,28,29 lead us to question whether publicly rebutting misinformation is useful and successful. These backfire effects are most likely to be found among audiences whose prior beliefs or political ideologies are threatened by the advocate28,30. For example, attempts to correct misconceptions about vaccination in an audience with low confidence in the safety of vaccination28 can ironically reinforce the misconception. The same effect occurred among US conservatives (who strongly object to governmental regulation regarding climate change) when there were attempts to debunk misinformation about climate change30; that is, when they received information that eventually might lead to regulation. This fear of governmental regulation has also been discussed as a cause for US conservatives distrusting scientists on the topic of vaccination14. These risks make it difficult for science advocates to decide whether they should participate in a public discussion at all19, potentially leading to the absence of advocates for science from a discussion (henceforth referred to as ‘advocate absent’).
It turns out that a combination of techniques probably works better than a single technique. That being said, the purpose of this study was to test two techniques: Topic rebuttal or technique rebuttal. Topic rebuttal is a simple concept: Rebut the claims using knowledge of the topic, by overwhelming the denier with good scientific evidence (cite and explain well-designed and executed experiments and studies) and/or explaining why the claims and evidence presented fail as science (e.g., weaknesses in design, cherry picking of evidence, misrepresenting evidence). Technique rebuttal is one of my favorites: Namely expose the deceptive rhetorical techniques being used (e.g., point out how the denialist narrative is a conspiracy theory, how the denialist is using deceptive or emotional rhetoric, etc.). I tend to like technique rebuttal because I’ve become a fan of “prebunking”; i.e., inoculating audiences against denialist arguments by exposing them beforehand to knowledge about the techniques of denial involving rhetoric, distortion and cherry picking of evidence, conspiracy theories, as well as why specific commonly used denialist arguments are wrong.
The experiment was carried out by six online experiments involving exposure to denialist arguments ± subsequent exposure to technique or topic rebuttals. Most of the experiments involved vaccination, but one involved human-caused climate change. Primary outcomes included, among others, intent to perform a behavior recommended by science (e.g., to be vaccinated). In all the experiments:
In all the experiments, participants first received an interview with a science denier. Participants were then randomly assigned to the following design, determining the rebuttal condition: 2 (topic rebuttal versus no topic rebuttal; between subjects) × 2 (technique rebuttal versus no technique rebuttal; between subjects) × 2 (time of measurement: before versus after the debate; within subjects) mixed design. Depending on the condition, a science advocate: was absent from the debate; responded to the denier by using topic rebuttal or technique rebuttal; or responded with a combination of both strategies (Fig. 1 provides an example of the materials used in Experiments 1–4 and 6).
And:
First, we analysed whether the denier influences the audience’s attitude towards and intention to perform the respective behaviour. Second, we analysed whether technique or topic rebuttal are effective strategies for reducing the denier’s influence and whether the combined strategy is more effective than the single strategies. Finally, we explored whether the influence of denialism and the effectiveness of rebuttal strategies are functions of the audience’s prior beliefs or political ideologies.
Without going into the weeds of the details, the authors did this experiment using nearly 1,800 subjects and found:
An internal meta-analysis across all the experiments revealed that not responding to science deniers has a negative effect on attitudes towards behaviours favoured by science (for example, vaccination) and intentions to perform these behaviours. Providing the facts about the topic or uncovering the rhetorical techniques typical for denialism had positive effects. We found no evidence that complex combinations of topic and technique rebuttals are more effective than single strategies, nor that rebutting science denialism in public discussions backfires, not even in vulnerable groups (for example, US conservatives).
And:
As science deniers use the same rhetoric across domains, uncovering their rhetorical techniques is an effective and economic addition to the advocates’ toolbox.
OK, so the results of this study suggest that not responding to science denial leads to bad outcomes, with science denialist arguments casting doubt in the audience about science. It is one study, and Kirsch is using it the same way as another “COVID contrarian,” Byram Bridle, did in 2023, falsely characterizing Timothy Caulfield’s refusal to debate him as being “not following the science.” Also, amusingly, the authors of the study include one huge caveat:
We acknowledge that in some situations contextual factors may still force the advocate to avoid participation (for example, the format of the discussion is not serious or personal safety is at risk31). However, with regard to the effectiveness of messages in conventional contexts, not turning up at the discussion at all seems to result in the worst effect. There may be one exception to this: if the advocate’s refusal to take part in a debate about scientific facts leads to its cancellation, this outcome should be preferred21,51 so as to avoid a negative impact on the audience.
In other words, let the antivaxxers find another patsy if they can. More importantly, the authors note:
Also, as can be seen in five of the six experiments (Fig. 2), the debate usually had an overall negative impact on attitudes.
In other words, even if we science communicators show up to “debate,” the overall effect of denialist arguments on the audience will usually still be negative towards science. All we can do in such debates is to mitigate this negative effect. That’s why the wag in me would turn it around on Kirsch and say that this study could be used to support not agreeing to a debate like his, because (1) refusing might mean that the debate never takes place and (2) if the debate takes place the best that the science communicator could expect to accomplish would be mitigate the damage of denialist arguments. If you accept the results of this study, you have to accept that the end results of such debates will not be to increase support for vaccination in the audience!
Moreover, although I don’t have any objective data to support this conclusion, my personal experience suggests to me that, if there are no science communicators present to “debate,” such events held by science deniers tend to attract almost exclusively those already predisposed to the science denial being promoted; i.e., they become just another echo chamber. That is, of course, not good. Echo chambers can radicalize further. However, the event will be much less likely to win any “converts” to the science denial cause than a debate. (I’d love to see someone study this question with rigorous methods, but just thinking about how to do such a study gives me a headache.)
Thanks, Steve. I haven’t reviewed this study since 2023! Also, I still find it amusing how antivaxxers and science denialists cite this study obsessively when in fact, depending upon your interpretation and proclivities, the study can be used to justify debating science deniers like Kirsch or refusing to do so.
But what about Dr. Mike. Clearly, he chose to debate, and he chose to do so in a format designed to disadvantage the one who is “surrounded.”
Dr. Mike vs. MAHA and RFK Jr fans.
Dr. Mike vs. 20 MAHA stans: Is it really about the science?
Before I start this part of the discussion, here again is the YouTube video of Dr. Mike’s debate with 20 MAHA supporters, for those of you who want to watch all nearly two hours of it. (Unfortunately, the video won’t let me embed it.) I also note that at the beginning of the video Dr. Mike notes that he’s surrounded by 15 RFK Jr. supporters. So much for 20, I guess. The exact number doesn’t matter that much, though.
My first observation was that Dr. Mike was quite effective at attacking specific points made by MAHA fans. Multipole examples stand out. For example, in an encounter with a woman who comes across as a typical “MAHA mom,” Dr. Mike pushes back against claims that doctors just want to keep their patients sick, so that they can make money. But first, he acknowledges her perception that doctors treat patients like “we’re kind of dumb” by conceding that this still can happen, before he pivots to asking the woman what she thinks of RFK Jr.’s longstanding (and not infrequent) claim that doctors are incentivized to keep their patients sick, asking, “Does that improve your relationship with your doctor or not?” The woman answers (at the 37:28 mark):
MAHA supporter: I would have to hear the context.
Dr. Mike: That’s the context.
MAHA supporter: Well, but I mean like the whole thing because one thing that I’ve experienced a lot with Bobby is that people get sound bites and they don’t hear the entire interview and the entire dialogue and then they peacemeal it and then they go that man.
Dr. Mike: Yeah, people people do do that with things, right? What’s a way that he could have been saying that in in a genuine way? Well, he in a genuine way.
MAHA supporter: Well, I think he was I I can’t speak for him. Sure. But my experience of Bobby is that he is a genuine, compassionate, kind, caring human and he takes his work very seriously and his commitment to ending chronic health disease and making sure children are healthy and adults are healthy is very very important to him. So, but if I heard that, I I probably would have been like, hm, I want to understand more about where he is coming from because it doesn’t line up for me that he would just generalize that all doctors are doing that. But in context, he might have been referring to something.
Dr. Mike: So, if one of the participants here that are surrounding us uh came up to me after the show and said that you’re a liar and everything you say is untrue, that perhaps could be said in a nice way?
MAHA supporter: Oh, well, I would probably think that maybe there’s something going on.
Dr. Mike (interrupting): Or called you a bad mother. Wouldn’t that hurt?
MAHA supporter: I’ve been a lot of things. It would make me think for a moment like, do you know me? Do you know what I do? Do you know how I am? Do you know my professionalism?
Dr. Mike: That’s how I feel as a doctor when Secretary Kennedy says that to me. He says we’re interested in people keeping people sick for profit. I don’t even take a salary to work as a doctor anymore. The YouTube stuff is so successful. I really am showing up for my patients pro bono because I want to help people. I’m here pro bono. I flew here myself for free because I want to give you the best information.”
One could look at this exchange as Dr. Mike skillfully using technique rebuttal, pointing out that RFK Jr. is positing, in essence, a conspiracy theory that doctors are in it to keep patients sick so that they make more money, although wisely he doesn’t use the term “conspiracy theory.” However, he goes beyond that to make an analogy to try to win the empathy of the MAHA mom by comparing RFK Jr.’s conspiratorial accusations about physicians to accusations that the woman is a bad mother, emphasizing that we physicians feel the same sort of emotional reaction as a woman accused of being a bad mother. After all, being a physician is central to our identities, just as being a mother is central to the identity of many women. Attacks on something that is central to our identity hurt, even more so when they’re not true. Then Dr. Mike skillfully lays the groundwork for what comes next:
Dr. Mike: And you said Secretary Kennedy is well researched.
MAHA supporter: Yes.
Dr. Mike: He goes on the podium and he says inaccuracies all the time. And look, everyone’s allowed to make mistakes. I’m not about gotchas. I’m not that guy. But when you say, back in the day, kids who had diabetes almost never existed and now one in three kids have diabetes, it’s a lie. One in 300 kids have diabetes. So like, how much do I as a YouTuber have to fact check Secretary Kennedy, the director of Health and Human Services, before he comes out and says, “I messed up. I need to tell you the truth, and I’m getting the numbers wrong so often.” When does that happen?
You and I know, of course, that the answer is: Never. RFK Jr. is an inveterate liar, and I am unaware of any instance when he’s admitted that a claim that he’s made, such as the one Dr. Mike discusses above, was in error. Of course, the question then becomes: Did Dr. Mike move the needle with this woman? The answer is, sadly, no. Instead of answering Dr. Mike’s question directly, she pivots to whataboutism and moves to COVID:
MAHA supporter: That’s a really good point. I don’t know. I guess we’d have to go back also to during COVID when they made these really outrageous statements that weren’t true. Either go back because because we there have been mistakes and…
Dr. Mike (interrupting): I have—I have multiple videos on my on my YouTube channel including podcasts where we talk about every mistake we made—I made—during the pandemic, because ultimately it’s about trying to figure out how to do better not to score political points.
And thus ended that exchange. To summarize, Dr. Mike employed empathy to try to get the woman to understand how doctors feel when RFK Jr. accuses them of all sorts of horrible things (like intentionally keeping patients ill so that they can profit and, although it wasn’t mentioned in the exchange above, intentionally making children autistic or even killing them with deadly vaccines, again for profit and ideology). Then he used his topic knowledge about diabetes to cast doubt on everything RFK Jr. says in a devastating fashion. It didn’t work to sway this woman, of course. A single exchange, not even one as effective as Dr. Mike’s, is likely to change the mind of a die-hard MAHA stan. At best, it might lead the MAHA stan to view Dr. Mike with less hostility. However, those watching who are not die-hard antivaxxers or MAHA stans could well be favorably influenced by Dr. Mike’s arguments, although the study cited by Kirsch could suggest that all Dr. Mike accomplished is to have decreased the negative effects on the acceptance of science-based medicine caused by watching 15 MAHA fans take turns going at Dr. Mike with science denying rhetoric.
There were a number of other instances where Dr. Mike, using a combination of topic rebuttals, technique rebuttals, and use of empathy to win over the audience, simply tore apart MAHA claims. In one instance, he explained why one of Dr. Peter Attia’s frequent claims is hugely overblown, which was particularly satisfying given that his opponent was acting as though he knew more than physicians. In another, when an opponent went on and on about doctors receiving pharma payments but couldn’t answer Dr. Mike’s questions about how we know how much money doctors receive from pharma. In response, Dr. Mike completely agrees with him that financial transparency is very important, after which he goes on to say, “Let me help you.” (I realize he was trying to be empathetic, but this was first class shade.) He then discussed the 2010 law called the Physicians’ Payments Sunshine Act, which requires disclosure of financial relationships between doctors and hospitals with pharmaceutical and device manufacturers. (BTW, if you look my name up, you’ll see that I receive no money from pharma. My name doesn’t even show up because, as the lookup site says, “Providers that have not received at least one payment in the past seven years will not appear in the search.”) He then reemphasizes that transparency is important and points out how there is no financial transparency in MAHA, as most MAHA influencers don’t have to report their financial relationships with, for instance, supplement companies or law firms suing for vaccine injury. A final example occurred when Dr. Mike agreed that opioid manufacturers got off far too lightly for having promoted opioid use and contributed to the opioid crisis.
You get the idea. A lot of what Dr. Mike did was not about the science. It was about values. It was about finding common ground in terms of values and then using that to point out where RFK Jr. is leading them astray. As you can see, though, it didn’t work with a hardcore “MAHA mom,” but it likely did at least make her less hostile to him, while making it more likely that observers might be swayed by Dr. Mike’s arguments.
Revisiting “debates” with science deniers
So what does this all mean? Am I a convert to the value of live public debates? Obviously, no. However, I am not as hostile to the idea as I used to be. Indeed, I think that Schmid and Betsch actually made a really good point near the end of their paper (you know, the one Kirsch cited so approvingly):
In relation to this, a third general take-home message is that advocates who take part in debates should not expect too much for their efforts. Therefore, facing deniers in public debates can be only one building block in the concerted effort to fight misinformation. Other recent approaches try to fight misinformation by pre-emptively providing laypeople with the ability to identify false information themselves20,21,52. For example, in a study conducted with Ugandan primary school children, researchers taught 10- to 12-yr-olds how to separate misconceptions about health treatments from facts52. Such educative approaches are in line with psychological research that attempts to inoculate individuals against misinformation20,21. The goal of inoculation is to make individuals aware of the arguments of denial before the actual information is obtained and to provide them with the ability to come up with counter arguments. An inoculated audience may be less susceptible to the arguments of deniers and the effects shown in the present experiments may be weaker in such an audience.
The way I look at it now is this way. Debating science deniers is a long run for a short slide that has a number of potentially devastating pitfalls. Because of that, it’s not for me, although I recognize that others might disagree. However, if you’re going to do it, you need to go in with your eyes open and train in a manner reminiscent of the pre-fight training montage from your favorite Rocky movie (metaphorically speaking, of course). Schmid and Betsch’s study appears to show that it makes little difference if you use one, the other, or both techniques; so it is best to go with your strengths, as they themselves state:
Still, being in a public debate with a science denier requires diligent preparation. It may seem like an endless universe of potential misinformation that is difficult to anticipate. However, analyses revealed that most topic arguments fall into five core categories and that deniers use the same five techniques to make those arguments appealing (Fig. 1; ref. 24). Hence, if they implement only one strategy (topic or technique rebuttal), advocates need to prepare only five key messages that address the core topics or techniques. It is important to note that we did not test all possible topics and techniques and that the effectiveness of the strategies may vary with specific topics and techniques. Nevertheless, training in technique rebuttal seems especially valuable as the techniques are the same across a broad range of scientific domains13,24, whereas the topics vary across domains (see Fig. 1 and Supplementary Fig. 1). Therefore, technique rebuttal is the more universal strategy in the fight against misinformation. Applying only one of the strategies may seem less complex; however, doing so successfully during an ongoing discussion will still require sufficient training.
I would tend to disagree with one thing though. This study aside, I would argue that, if you’re going to go into such a “debate,” you need to develop a high level of both topic and technique knowledge, even if you plan on primarily using technique rebuttal. Technique rebuttal can only take you so far in combatting a Gish gallop. You need to be able to point out at least one or two glaring errors in the Gish gallop in order to bolster your credibility when you go for the technique rebuttal pointing out that your opponent is doing a Gish gallop.
As we head into 2026, I think it’s important for those of us trying to counter the tsunami of misinformation that is now coming from “inside the house,” so to speak, given that the federal government is now controlled by antivaxxers and science deniers to play as much as possible to our strengths. It’s equally important to take heed to Harry Callahan’s advice in Magnum Force and know your limitations. In other words, I’m happy to let Dr. Mike do his thing on YouTube, for instance, and podcasters to do their thing to communicate science (I’m a pretty good podcast guest, but have no clue how to do my own podcast), while I do my thing here.
Finally, perusing the literature, the depressing realization that I come to is that we desperately need better data, experiments, and trials of techniques of countering misinformation and disinformation. I’ve long resented how little in the way of academic reward engaging in this work, either as a hobby (me) or as an academic pursuit yields. I’ve referred to how much of the scientific community looked down on Carl Sagan’s work last century, and that disdain, although it’s greatly abated since I started two decades ago, is still there. Unfortunately, with the cranks firmly in control of the federal government and, in particular, the NIH, funding for misinformation research, which was always inadequate, is now nearly nonexistent. Yet the need has never been more acute, leaving us in the meantime to do the best we can with what we have and to do what each of us does best.


The “We’re Not Allowed to Question This” Gambit
Anyone who truly values open and rational discussions about controversial subjects need to be cleared-eyed about where the threat to such dialogue is coming from.
Jonathan Howard on May 8, 2026
Link
“Our inability to remain open and engage in rational discussions about controversial subjects may be eroding public trust in science.”
Dr. Akiko Iwasaki is the Sterling Professor of Immunobiology and Professor of Dermatology and of Molecular, Cellular, and Developmental Biology and of Epidemiology at Yale University. She is an infinitely more accomplished researcher than I will ever be, and to the extent that I can divine such things, not someone who acts in bad faith or with nefarious motives.
However, she recently penned a problematic essay, one that reminded me of my prior article Debating How to Debate Instead of Actually Debating. That article discussed doctors who spent all their time and energy debating the process of debating, while refusing to simply debate. For such doctors, some of whom were vulgar and threatening themselves, serious issues of science and medicine were subsumed to comparatively trivial concerns of manners and decorum. “Every moment spent debating who can say what to whom and how and where they should say it, is time not spent talking about doctors who repeatedly underestimated the virus and successfully campaigned to ensure tens of millions of children and young adults contracted it before they were vaccinated,” I wrote.
Dr. Iwasaki’s entry into this sad genre was titled Freedom of Scientific Inquiry: Reclaiming Space For Controversy. It was published in Nature Immunology and it began by saying:
No matter how small the possibility, or how inconvenient the topic, scientists must maintain the humility to acknowledge that we could be wrong. Our inability to remain open and engage in rational discussions about controversial subjects may be eroding public trust in science.
At first blush, this is banal, unobjectionable pablum, like saying “We need more RCTs.” After all, who could oppose “open and rational discussions”? Not me. All of my articles, including this one, are written in that spirit. However, as with doctors who glorified RCTs not to honestly advance research but rather to spread doubt and rage about COVID mitigations, Dr. Iwasaki’s seemingly reasonable call for “open and rational discussions” also sought to advance a particular narrative.
Indeed, Dr. Iwasaki curiously singled out two classes of scientists for whom we should have special sympathy at this fraught moment- those who spoke about potential vaccine harms and those who claimed SARS-CoV-2 leaked from a lab. She wrote:
Here, I discuss the need to reclaim protected space for rigorous inquiry into controversial questions — including the possible adverse effects of vaccines and the origins of the SARS-CoV-2 pandemic.
How did these poor scientists suffer? According to Dr. Iwasaki, for the most part, they had their feelings hurt on social media and weren’t treated with sufficient deference in newspapers. She wrote:
To prevent creating unnecessary fear of vaccines, most immunologists prefer not to discuss rare adverse events after vaccination. Even before the pandemic, discussing and publishing any negative effects of vaccines was viewed as ‘anti-vax’, somehow betraying the field of immunology…
In the wake of the pandemic, polarization and public mistrust of science are at their highest levels in decades. The pandemic did not create this mistrust but rather exposed and intensified existing fractures, fuelled by rapid amplification on social media.
Although many external factors contributed to this, we scientists are not blameless. I have witnessed and even succumbed myself to the temptation to be certain. Often, scientists are ‘100% sure’ of something, and anyone questioning that position is treated as an idiot. “Leave the discussion to the experts”, “stay in your lane” and “don’t listen to armchair immunologists” are phrases used to dismiss and ridicule anyone who questions mainstream views. Granted, social media platforms are not designed for rational and respectful discourse. Yet the “don’t question scientists” approach is a sure way to alienate people and cause them to lose faith in science, and it is also antithetical to the scientific process. Science is an iterative process in which con-cepts and hypotheses are tested by several groups, building consensus that is essential to understand how things work and to make progress. However, consensus, when combined with ideological or political biases, can become a constraint, evolving into an orthodoxy that suppresses alternative viewpoints…
However, discussions about the origin of the virus have been highly contentious, with those who suggested that it might have originated from a possible accidental lab leak often labelled as conspiracy theorists or racists and ostracized from the scientific sphere. Scientists who were investigating and communicating about the origin of SARS-CoV-2 encountered harassment and dismissal. Reporting in the mainstream media did not help either, with statements such as “Most scientists believe …” being based on interviews with a few, possibly the most vocal, advocates of a particular view.  However, allowing for an open inquiry into inconvenient questions surrounding SARS-CoV-2 is important — not to assign blame but to refine biosafety and preparedness frameworks.
It is possible that some scientists were subject to harassment and dismissal only because they promoted the lab leak, which would be unacceptable, however I couldn’t find any instance of anyone losing their job for this reason. I will update this article if I learn of any such cases, but Dr. Iwasaki did not provide the names of scientists she felt were so mistreated. I honestly have no idea who exactly she is talking about, and with one exception, she provided no references to back up any of her claims.
“Scientists are unable to freely inquire about the risk of post-vaccination syndrome without being labelled as ‘anti-vaxxers’.”
Indeed, Dr. Iwasaki provided just a single reference, to a preprint she posted in February 2025 titled Immunological and Antigenic Signatures Associated with Chronic Illnesses after COVID-19 Vaccination. It is worth reviewing this study as it was the impetus behind Dr. Iwasaki’s recent editorial.
The preprint used a machine learning model to describe a condition she termed “post-vaccination syndrome (PVS)”. In her Nature editorial Dr. Iwasaki described PVS and the reaction to it thusly:
PVS is a poorly characterized constellation of heterogeneous, non-specific symptoms that lacks an agreed-upon clinical definition, diagnostic criteria or standardized measurement. Owing to the stigma and judgement that surrounds this condition, scientists are unable to freely inquire about the risk of PVS without being labelled as ‘anti-vaxxers’.
Unsurprisingly, this poorly characterized constellation of heterogeneous, non-specific symptoms attracted a lot of attention. While anti-vaxxers naturally celebrated and misrepresented the preprint, leading to justified and valuable pushback from Dr. Iwasaki, many legitimate scientists identified numerous potential problems with its methods and study population. They did so in a civil, professional manner, exactly the sort of open and rational discussion Dr. Iwasaki claims to value. Some examples are below:
    • No, Covid-19 Vaccines Don’t Suppress Your Immune System—Here’s The Science
    • Yale Covid Vaccine Syndrome Study
    • Fact Check: Us Study Does Not Suggest Long Covid Is A Vaccine Injury
    • No, A New “Study” Doesn’t Prove Covid Vaccines Are Unsafe
    • How A Yale Study Got Twisted Into An Anti-Vaccine Talking Point
    • Winner of mRNA Nobel Prize Says ACIP member’s Claim That Covid Vaccines Persist is ‘Absolutely Impossible’
    • ‘Post-Vaccination Syndrome’: A Dubious Diagnosis That Could Do Harm
None of these critics labelled Dr. Iwasaki an “anti-vaxxer”, though that description seems to fit several of her co-authors. While I encourage you to read all of these critiques of her preprint, according to the last one, by Dr. Adam Gaffney:
Two of the co-authors, described as “independent researchers,” are part of the organization React19, a vaccine injury advocacy organization engaged in Covid vaccine injury litigation. One of these individuals is personally a vocal and high-profile litigant who supported Robert F. Kennedy Jr.’s nomination as Health and Human Services secretary and who published a vaccine-injury testimonial blurbed by Kennedy. A third co-author has made dubious claims about chronic Lyme disease and has endorsed supplement use without evidence as a treatment for long Covid. A fourth has speculated online that another potential co-infection culprit in PVS might be “dormant Bartonella infection,” a real but rare pathogen that for unclear reasons has emerged in fringe medical circles as a commonly invoked tick-borne “Lyme coinfection,” as described by Centers for Disease Control and Prevention. These are red flags, to say the least.
Apparently, these significant conflicts of interests were not declared in the initial version of the preprint, and Dr. Iwasaki also promoted the study on social media by linking to a press release from Invivyd, a pharmaceutical company that makes monoclonal antibodies. These are just some of the many concerns with her study, which remains unpublished and unreplicated to this day.
“We need to leave room for people to have good faith disagreements and questions without inappropriately smearing them as ‘anti-vaxxers’.”
Obviously, Dr. Iwasaki is not wrong about how scientists should be treated when they discuss controversial topics. Anyone with potential evidence of vaccine harms or a lab leak should be given a respectful hearing to make their case.
Unfortunately, I know from experience that anyone who waded into pandemic controversies received a deluge of vitriol from all directions. There was nothing unique about Dr. Iwasaki’s experience, which is not to excuse or minimize any spiteful comments towards her. Anyone who simply insulted her should absolutely apologize.
However, while the doctors who hurled an endless stream of juvenile insults at me never engaged with my content, as the thoughtful essays above show, that was not the case for Dr. Iwasaki. And so, the problem with her Nature editorial isn’t that she advocated for open scientific debate, the problem is that she implied it didn’t happen- “scientists are unable to freely inquire about the risk of PVS” “allowing for an open inquiry into inconvenient questions surrounding SARS-CoV-2 is important.”
In reality, Dr. Iwasaki got funding for her study on PVS, she conducted the study, and posted her preprint. She then discussed PVS on social media, Yale publications, the New York Times, as well as popular podcasts and YouTube channels. Pharma companies even invited Dr. Iwasaki to join their study groups. Today, she is writing about PVS in Nature. Not bad for a preprint that has yet to be published in a peer-reviewed journal.
All of this shows that scientists who discussed vaccine harms and COVID’s origins weren’t systematically suppressed. While some scientists, bad faith actors who also claim to have been “silenced and censored,” exploited these controversies to advance their political agenda, the truth is that countless scientists discussed them freely for years. There was no shortage of scientists and doctors willing to talk about vaccine side effects and the origins of SAR-CoV-2.
I speak from experience here as well. I also discussed vaccine flaws extensively in my books, blogs, and podcasts. “Care should be taken to not minimize these cases,” I said in my first essay about vaccine-myocarditis nearly 5 years ago. “The COVID vaccine is far from perfect,” I wrote in my most recent essay here. It’s entirely possible that my privileged demographics shielded me from nasty insults for writing such things, but no one ever told me to stop or called me an anti-vaxxer. I also don’t recall anger at the scientists who linked the J&J vaccine with fatal blood clots, even though this led to the vaccine being taken off the market.
Scientists who provide solid evidence of vaccine harms are generally treated respectfully and professionally, as they should be, and many years before Dr. Iwasaki wrote her editorial, I too opposed the use of “anti-vaxxer” to discredit people who raise valid concerns about them. I wrote an entire article titled I Won’t Call Dr. Paul Offit an Anti-Vaxxer, and in a separate article I wrote:
Though most vaccines are overwhelming safe and effective, their imperfections cannot be ignored or wished away… We need to leave room for people to have good faith disagreements and questions without inappropriately smearing them as ‘anti-vaxxers’.
Similarly, those who claim SARS-CoV-2 came from a lab have had ample opportunity to state their case, including at academic conferences, though in contrast to scientists who support a zoonotic origin, none of them have ever presented affirmative evidence of a lab leak in the scientific literature. However, Dr. Iwasaki implies that while this evidence exists, it wasn’t “allowed” to be heard or it wasn’t taken seriously only because journalists had the audacity to write “most scientists believe” the virus had a zoonotic origin in their articles on the topic.  
So sure, we can all endeavor to be nicer to people who discuss potential vaccine harms and promote the lab leak. Our goal should always be to engage with their arguments and never to deliberately hurt their feelings. Conversely, scientists should engage with good faith criticisms of their work, not reflexively brush them aside by falsely labeling them as personal “attacks” and “hit pieces”. As Ralph Waldo Emerson said:
Let me never fall into the vulgar mistake of dreaming that I am persecuted whenever I am contradicted.
Not everyone who criticized Dr. Iwasaki’s preprint simply smeared her as an anti-vaxxer. Many scientists raised valid concerns about the robustness of her methods and thus the credibility of her results. Instead of questioning their manners and motives, hopefully Dr. Iwasaki will soon respond to the substance of their critiques.
“Vaccines are the ONLY drug you’re not allowed to question.”
The “we are not allowed to question this” gambit, as I will call it, isn’t new. Disingenuous anti-vaxxers have long pretended that questioning vaccines was “forbidden”. In their minds, only they were brave enough to ask the tough questions, and they were unjustly persecuted for doing so. Listen below to a compilation of disinformation luminaries Joe Rogan, Tucker Carlson, Robert Kennedy, and Cheryl Hines question vaccines, while at the same time saying vaccines are the ONLY drug you’re not allowed to question.
When employed by respected scientists, however, the “we are not allowed to question this” gambit validates anti-vaxxers’ delusions that scientists are afraid to level with them about the risks of vaccines. By erasing years of discussions and debate on vaccines and COVID’s origin, the gambit also becomes another example of the COVID Amnesia Project, which feeds the absurd victim narrative and hero complex of the pandemic’s worst disinformation superspreaders, the people now in charge. 
“He opened it to find a half-dozen federal agents carrying guns and wearing tactical gear, including bulletproof vests.”
To be clear, these are dangerous times for scientists, and social media insults are the least of their concerns. However, the threat is not directed against scientists who spoke of vaccine harms and promoted the lab leak. Rather it comes from such forces, and it is directed against pro-vaccine scientists and those who reject the lab leak.
Drs. Marty Makary and Jay Bhattacharya, for example, routinely treated rare, temporary vaccine side effects as a fate worse than death and promoted the lab leak in propaganda videos. Dr. Bhattacharya even smeared Fauci as “the number one anti-vaxxer” 5 years-ago because Fauci wisely urged caution after vaccination. Today, Drs. Makary and Bhattacharya are not oppressed and silenced. Their embrace of anti-vaxx disinformation and the lab leak is precisely what catapulted them to the pinnacle of the medical establishment. In that role, they are not just still spreading anti-vaxx disinformation, they are using their vast power to censor and purge scientists who don’t share their anti-vaxx agenda.
Other scientists are paying an even bigger price for not endorsing the lab leak. According to one horrific article, Guns and Bulletproof Vests: How Federal Agents Arrested Fauci Aide:
When federal agents came to the home of David Morens on Monday with an arrest warrant for allegedly concealing federal records related to the debate about the origin of the COVID-19 pandemic, they behaved as if the 78-year-old retired scientist was a violent criminal.
Science has learned that Morens, an influenza researcher who worked at the U.S. National Institute of Allergy and Infectious Diseases (NIAID) until 2022 and was an aide to its former director, Anthony Fauci, was having his morning coffee at his Chester, Maryland, home when he heard loud pounding on the door. He opened it to find a half-dozen federal agents carrying guns and wearing tactical gear, including bulletproof vests, according to two sources who spoke with him but asked not to be identified. Another team of officers stood in the distance and observed, as did neighbors.
The agents did not harm Morens, but took off his pants and shirt, handcuffed him, and drove him 65 kilometers to the U.S. District Court in Greenbelt, Maryland, where he was fingerprinted, photographed, and jailed.
Meanwhile, Dr. Peter Hotez routinely received death threats and was stalked at home by anti-vaxxers. Fauci’s children needed bodyguards, MAGA doctors consistently call for his arrest, and he still receives vile threats, including from top government officials who blame him for the lab leak. I mention these outrages not to imply that Dr. Iwasaki supports them in any way- I am certain she does not- but rather to contrast their experiences with scientists who were merely called names on social media.
Because with all that’s happening in the world of science and medicine with MAGA/MAHA in power, we don’t have to give credence to anyone’s fantasy that scientists who discussed vaccine harms and promoted the lab leak are a persecuted minority, under serious threat right now. To validate this absurdity gives ammunition to powerful, dangerous, vengeful people, the people with power, the people who are currently censoring, purging, and aggressively arresting elderly scientists today.
Anyone who truly values open and rational discussions about controversial subjects need to be cleared-eyed about where the threat to such dialogue is coming from.

   The clinician-scientist: Wearing two hats
David Gorski
Link
About a week ago, Tim Kreider wrote an excellent post about the differences between medical school training and scientific training. As the only other denizen of Science-Based Medicine who has experienced both worlds, that of a PhD and that of an MD, and as the one who two decades further along the path than Tim (give or take a couple of years), his musings reminded me of similar musings I’ve had over the years, as well as emphasizing yet again something I’ve said time and time again: Most physicians are not scientists. They are not trained like scientists; they are trained to apply scientific knowledge to the care of their patients. That’s what science-based medicine is, after all, applying science to the care of patients. Not dogma. Not tradition. Not knowledge of antiquity. Science.
Leave dogma, tradition, and “ancient knowledge” to practitioners of “alternative medicine.” That’s where they all belong. Whether you want to call it “alternative medicine,” “complementary and alternative medicine” (CAM), or “integrative” medicine (IM), it rarely changes and almost never abandons therapies that science finds to be no better than placebo, whereas scientific medicine is, as it should be, ever changing, ever improving. I’ll grant you that the process is often messy. There are often false starts and blind alleys, and physicians are all too often reluctant to change their practices in response to the latest scientific findings. We sometimes even joke that for some practices, it takes the supplanting of one generation of physicians with a new generation to get rid of some practices. But change does come when the science and evidence are there. Indeed, for example, in response to evidence that a bacterium, H. pylori, causes duodenal ulcers, medical practice changed in a mere decade, which is about as fast as anyone could do the science and clinical trials to show the validity of the new concept. Although CAM practitioners like to hold up the example of Barry Marshall and Robin Warren, the researchers who discovered that H. pylori causes most duodenal ulcers, as an example of how researchers with radical ideas are ostracized, but that story is largely a myth, as our very own Kim Atwood showed.
The application of science to medicine is a difficult thing. It takes basic scientists and clinicians, but the two of them exist in different worlds. Or so it often seems. That’s why some individuals seek to straddle both worlds. Tim is one such person. So am I. Unfortunately, most people don’t understand what we do very well. We wear two hats. In my case, I’m a surgeon, and I’m a scientist. In Tim’s case, he’s a scientist and a physician, but he doesn’t yet know what kind of physician he will end up being. At the risk of sounding somewhat arrogant, I believe that we, and others like us, represent an important element in bridging the gap between basic science and clinical science, in, essentially, building a more science-based medicine.
By saying that people don’t understand very well what clinician-scientists doI don’t mean that people don’t understand what it is that a doctor or a surgeon does. Everyone knows that a doctor takes care of patients and that surgeons operate on patients, after all. I also don’t mean that they don’t understand what a basic scientist does, although most of them do seem to have some sort of an odd picture of us in a lab with test tubes and brightly colored solutions bubbling in the background, the same misconception that is common about pure basic scientists. In fact, I would go so far as to say that even most doctors and most basic scientists probably don’t understand the difficulties of combining a clinical career with laboratory research, because those of us who try to do both are in a distinct minority. The vast majority of doctors do only clinical practice, and basic scientists, not having gone to medical school, will never have the responsibility of caring for patients. Even among the minority of doctors who actually do research, the majority do mainly clinical research (testing new therapies in clinical trials, for example), and that is what most people think of when they think of doctors doing research.
Combining a laboratory research career with a clinical career has always been difficult, but these days it’s become a Herculean challenge. One aspect of biomedical research not understood by most lay people is that basic scientists in academia doing research are usually expected to pay much (usually at least 50%) of their own salaries and all of their laboratory overhead through grant support, preferably from the NIH through the “gold standard” grant known as an R01. In smaller colleges, there are often faculty whose primary responsibility is to teach, but the big universities tend to require research and grant funding to support that research. If young faculty members don’t demonstrate the ability to compete for such grant support, they don’t get tenure. Usually, upon being hired fresh out of their postdoctoral fellwoship, they are given a startup package with enough support to keep their lab going for perhaps three or four years. After that, they are expected to be able to fund their own laboratories entirely through external funds. The bottom line is: If they can’t achieve that in the time allotted, they don’t get tenure, and often when they fail to get tenure they are shown the door. It’s a very Darwinian system. Yes, there are other requirements for tenure, such as peer-reviewed scientific publications of sufficient number and quality, science good enough to win the respect of peers, and evidence of teaching talent, but the bottom line is: If you can’t attract enough funding, the other stuff will hardly be considered.
Now, consider the clinician-scientist. We have to compete for grant money with the same hungry basic scientists who can spend the vast majority of their time doing research. Worse, most of the reviewers in the study sections that decide which grant applications get funded are basic scientists. Yet, a large proportion of our time is taken up caring for patients. Those of us who have not undergone formal Ph.D. training may not have adequate experience in formulating a research plan in a logical and compelling narrative that can convince a study section that (1) we have a reasonable hypothesis; (2) we have evidence to support the hypothesis; and (3) we can formulate a scientifically credible plan to study that hypothesis. And we have to do this while devoting a significant part of our time to patient care. I am fortunate enough as a clinician-scientist in that I have a position where I can spend more time in the lab and office than I do taking care of patients, but my patient care responsibilities are still something that take up a lot of time. Consequently, some basic scientists look at doctors (especially surgeons) with condescension. (“Oh, isn’t that precious! He’s trying to do serious research!”) Unfortunately, occasionally, that condescension is deserved–but it is not deserved nearly as often as some basic scientists seem to think it is.
There was a time twenty or thirty years ago when clinician-scientists could be supported by “slush” funds derived from the clinical income of the department in which they worked. This was especially true of surgery departments, which were often revenue machines. Those days are long gone, dead and buried. Academic medical centers now have to live in the same world as private hospitals from a financial standpoint, and managed care companies are not willing to pay them more, even with the increased overhead and inefficiencies that come from training medical students and residents. True, Medicare does pay academic medical centers a per-head sum for such training, but that only covers direct costs. It doesn’t cover the hidden costs that come from the infrastructure that it takes to train residents and the inefficiencies that training residents involve. For instance, most surgeons could get a lot more operations done in a week if they didn’t have to carefully and slowly take residents through them. That would mean more revenue. But we academic surgeons have made a conscious decision that we believe that training the next generation of surgeons is of the utmost importance, and we derive many rewards from it. For example, there’s nothing like seeing a trainee achieve success. It reflects back on everyone who trained him or her.
Be that as it may, between decreased reimbursement and the increased expenses involved running an academic department, there is often little or nothing left over for research after paying the bills–if the department is lucky enough and well-managed enough not to be in the hole. In fact, academic physicians are generally expected to bring in enough clinical revenue to cover their own salary and expenses. Given that the actual collection rate for what is billed may be only $0.25 to $0.40 on the dollar (or even less), for a surgeon, covering one’s own salary, one’s secretary’s salary and overhead, office overhead (yes, the University charges office rent), and other expenses can require billing for as much as $500,000 to $1 million a year–or even more. Busy clinicians can pull this off. Part-time clinicians don’t have a prayer, particularly if they have the added overhead of a laboratory. That means, to support clinician-scientists, an academic department must explicitly make the commitment to support research in a bit of an almost “communistic” endoavor. Busy clinicians in the department must buy into the vision and be willing to subsidize the development of new physician-scientists with some of the fruits of their own clinical productivity, accepting, in essence, less income than they could make otherwise in return for the benefits of belonging to an academic department and practicing at an academic medical center. That is, in essence, the compact. The other part of the compact is that physician-scientists usually end up accepting considerably less salary than they could make in private practice (or even if they were a busy academic clinician) in order to have enough protected time to do their research. This is not an inconsiderable hurdle for a new clinician-scientist to face in these days of medical students finishing with a quarter of a million dollars of student loan debt, which is why MD/PhD programs funded by the NIH are so important.
But what about grant support? That can help, but a scientist can only ask for salary support for the percentage of his time that is devoted to the research project for which grant support is being requested. Consequently, if you are 50% clinical, there is no way you can ever support your entire salary, as basic scientists can, with grants. The most you can ever hope to support is 50%. Rare clinician-scientists can support maybe 70% of their salary if they are (1) only 30% clinical and (2) good enough at getting funded to fund the 70% effort they are putting towards research. It’s worse than that, though. Most clinicians make an academic base salary, plus a salary that comes from clinical revenue generated by the department. Grant support can only ever be applied to the academic base salary. Consequently, clinicians can rarely cover more than 20-40% of their total salary with grant support. Again, even with an NIH grant, almost no clinician-scientist can cover his own overhead through clinical activity and grant support, even very successful ones. It would take several hundreds of thousands of dollars a year in government grant support, which includes around 50% or so added to the grant for institutional overhead, to have a shot at doing it. The benefits physician-scientists bring to the department are usually not financial; rather they include intellectual benefits and the prestige that an active and successful research program brings. Again, not all departments can afford such benefits in this climate of diminishing reimbursement.
Of course, as clinician-scientists, we do have one advantage over basic scientists in one area. We understand the clinical management of the disease we’re studying in an up-close and personal way that the basic scientist can never match because they don’t take care of patients. We deal with patients with the disease and watch the course of the disease every day. We know the deficiencies in present therapies and issues needing more research attention in a way that basic scientists have a very hard time matching. We make observations about the disease that basic scientists have an even harder time making. Indeed, the flip side of the condescension basic scientists all too often heap upon clinicians trying to do basic and translational research is the similar condescension far too many clinicians heap upon the way basic scientists tend to focus on molecular and cellular mechanisms rather than practical results. We want new therapies now because patients are suffering and dying now, and are often impatient with the leisurely pace (or leisurely it seems to us) at which pure science basic scientists move. This is not an insignificant factor, given the reorientation of the NIH in recent years towards research that is likely to lead to treatments sooner rather than later. Nonetheless, overall, the forces arrayed against the success of clinician-scientists are formidable indeed, and bridging the differences between the cultures of basic scientists and pure clinicians is not the least of them.
Perhaps the most formidable challenge for a clinician-scientist is finding enough “protected time” (time with little or no patient care responsibility that is “protected” for research). One of the most challenging tasks a young clinician-scientist has is to protect his research time from the inevitable intrusions of patient care. If he doesn’t, then one day he will wake up to realize that he hasn’t been in his laboratory for anything other than brief visits for over a month; that his entire schedule has been taken over by patient care demands; and that his lab is about to run out of money because he never had time to oversee the production of enough preliminary data to write a competitive grant application. Having adequate protected time is impossible unless a young faculty member’s chairman understands the need for protected time and is willing to help protect that research time. Sometimes that means laying down the law that new patients beyond a certain number will have to be seen by other surgeons. Without the chairman’s support, even the best effort to protect research time will likely fail.
I’ve seen it with colleagues and friends over the years. Driven by the unfortunate financial reality of academic surgery today, their chairmen or division chiefs, while paying lip service to the need to provide protected time for research, would demand more and more clinical revenue, which means seeing more and more patients. For a while, the clinician-scientists would try to make up for steady infiltration of their research time by patient care responsibiliteis by working more and more late nights and weekends, but eventually something had to give, and almost inevitably it would. In these cases, what almost always “gives” is the research. The clinician-scientist concludes that he is forced to give up research and–usually–become primarily a clinician. Sometimes the rare surgeon-scientist with an exceptional devotion to and talent for research will become a pure basic scientist, but they can only do that if they are so successful at publishing and applying for grants that their labs are, in essence, awash in grant money. I’ve been extraordinarily fortunate so far in that all my bosses and the institution have done their best to help me protect my research time, but I have no illusions that, were I ever to lose all my grants and show no promise of obtaining more, my current institution (or any institution) would cut me slack for very long.
In some ways, I think that surgeons trying to do research have it worst of all. In the medical specialties, it is often possible to arrange schedules so that doctors doing research only have concentrated clinical duties one or two months out of the year, with the rest of the year serving as protected time. In medicine, pediatrics, and other specialties, this is known as being “on service” and usually involves being the attending who is responsible for an inpatient service for the month. Once the month’s up, the inpatient clinical responsibilies end, leaving only however much outpatient clinical responsibility was agreed to. For busy clinicians, this can be a lot of outpatient responsibility, but for clinician-scientists, it is often designed to be only one day of clinic a week or even less.
In surgery, such huge chunks of protected time are rarely possible for several reasons (outside of V.A. or county hospitals, and even there such huge chunks of protected time are rapidly disappearing). First, surgery is personal. Patients don’t want just any surgeon operating on them. They usually want a specific surgeon that they’ve come to trust. Consequently, the clinical burden is ongoing throughout the year, leading to us trying to protect days at a time, rather than weeks or months at a time. Second, even if we could arrange our schedules the way medical doctors do, our specialties are skill- and task-oriented. We would risk the atrophy of our operative skills if we were only to operate intensively one or two months a year. (Indeed, clinical surgeons sometimes look down on surgeon-scientists as not having the same level of surgical skills they do.) Finally, surgeons have a very special relation to their patients. If a patient I operate on has a postoperative complication, I’m going to take care of it, unless for some reason I’m on vacation or out of town (and for some surgeons, even those are not always barriers to taking care of their own). It doesn’t matter if I’m on call or not, if it’s the middle of the night or not, or if it’s during the weekend or not. That’s just the nature of surgery as a specialty. Although my surgical specialty has relatively few emergencies, for other surgical specialties (for instance, GI surgery), just this aspect of surgery alone can make a productive research career extremely problematic. The bottom line is that we are expected to be just as good as surgeons who are pure clinicians while devoting only part of our time to surgery and just as good at laboratory research as basic scientists–again, while dedicating only part of our time to it, usually a lot less than basic scientists can. We rarely succeed at both to an equal degree.
I realize that basic scientists will retort that they have other responsibilities, too. They teach, sometimes a lot. They sit on committees. They have bureacratic responsibilities that take them away from their research every bit as much as surgery. Well, not quite. If a basic scientist fails to fulfill his teaching or administrative duties or doesn’t fulfill them well, no one is likely to die or to suffer a complication or additional pain. Patient care often can’t be put off, at least not for long. Unlike teaching, it can’t always be predicted or scheduled. That’s the difference. It’s possible to succeed as basic science faculty while being not so good at other responsibilities, like teaching. My own experience with several professors during graduate school bears this observation out. There were a few professors who were highly successful in their research, with many publications, international reputations, and oodles of grant money. Unfortunately, some of them were awful in the classroom. There were even more faculty who were just OK in the classroom, but–again–ran successful laboratories.
I realize that what I said may sound arrogant to some or as though I am denigrating the difficulties basic science faculty face. I assure you that it is not and I am not. Not having much experience in teaching classes, I would probably have an absolutely hellacious time at first learning to become a competent classroom teacher (as opposed to a clinical teacher, which is different) if I ever tried to be straight basic science faculty. However, it’s just the nature of the beast. You can’t afford to be bad or even mediocre as a physician, regardless of whether or not you are a scientist as well. You just can’t. If you are bad or mediocre, you have no business treating patients. You have to be at least competent as a physician or surgeon, and preferably you should be excellent. It’s also more difficult in a highly technical specialty, like surgery. Practice makes a difference in highly technical skills, unless you happen to be one of those lucky surgeons who is just naturally gifted, and, even then, practice is what turns the technically gifted surgeon into a superstar.
Maintaining one’s medical and surgical skills as a part-time physician is perhaps the most difficult challenge facing the surgeon-scientist or the clinician-scientist. The only way many of us manage it is to focus their clinical practice like a laser very tightly on one specialized area that interests us and about which we’re passionate, which is what I have done. That’s the practicality of it. The necessity for competence when you don’t get as much experience and practice as a straight clinical surgeon almost always mandates focus. Even then, it remains really difficult. As a part-time surgeon, I don’t do nearly as many cases as a full-time surgeon does, and, as long as I continue to run a lab, I never will. That is another reason by my practice has to be tightly focused if I want to do right by my patients and still have the opportunity to do research.
It cuts both ways, too, although that’s usually less of a consideration compared to the basic science faculty with significant teaching responsibilities. I’m in essence a part-time scientist as well. I can’t devote the same amount of time to writing grants, papers, supervising the lab, or even doing experiments that most basic science faculty can and do. There just isn’t time. But perhaps the biggest issue that cuts both ways is the literature. In essence, physician-scientists have twice as much scientific/medical literature to deal try to keep up with. We have to keep up with the medical literature involving our specialties and the scientific literature involving our research, and we have less time to do it, to boot. No wonder I never feel as though I’m on top of the surgical literature. No wonder every so often I get blindsided by a paper related to my research that I never noticed, probably more so than basic scinetists do. That’s why another essential practical necessity for success is to have good people working for you and good collaborators working with you. Although this is true for any researcher, for clinician-scientists it’s doubly true: One bad hire can destroy your lab’s productivity and even ultimately your lab. You need people who can work with little supervision, and you need collaborators who can help you out with the more arcane basic science that you aren’t trained in. In return, you offer your collaborators your clinical understanding of the disease process and, if you happen to be a surgeon, one of the most precious resources of all for biomedical research: access to human tissues.
Finally, perhaps the key difference between being a basic scientist and a clinician-scientist is predictability. The basic scientist, even the one who has significant teaching responsibilities that take up a big chunk of his schedule, has a much more predictable schedule. Patient care can be made somewhat predictable, but emergencies will always occur, the number and frequency of which depend upon the specific specialty. There will always be calls on weekends or in the middle of the night. Disease doesn’t respect weekends or nights. Moreover, the natural tendency is for patient care responsibilities to grow slowly and inexorably. In essence, a surgeon can be a victim of his own clinical success, and his success in the clinical realm can negatively impact success in the scientific realm. I know one surgeon who has become so popular that every time he tries to cut back his practice in order to engage in more academic purusits, there is a backlash from his referring physicians, who start calling him to ask him if he can make an exception “just this one time” for this patient or that.
The answer is always yes.
Let’s look at a few real-life scenarios that illustrate the conflict between doing bench research and taking care of patients. Scenario number one: You are in the middle of a big experiment when the E.R. calls about a patient you recently operated on, who has returned intrabdominal sepsis. It turns out that your anastomosis has broken down, and the patient needs urgent surgery to fix the problem. However, if you take off to do the surgery, your experiment will be ruined, wasting days of work and hundreds of dollars worth of reagents. The choice is really no choice at all; you must take care of the patient and trash your experiment. You could ask one of your partners to deal with the complication, but they have patients of their own to deal with. They’re all either in clinic or in the operating room, anyway. Besides, it’s your patient’s complication. You need to deal with it, because that’s how surgeons roll.
Scenario number two: You have a grant due in a few days. You thought you had planned well, having canceled your clinic for that week several months prior and told the schedulers not to schedule any operations that week, in order to allow maximal time to finish the grant. However, there is a patient that your boss tells you that you must see and take care of now. Alternatively, one of your top referring physicians is asking you to see this patient as soon as possible. One more variation: The patient is a V.I.P. that the CEO of your hospital is asking you to see.
Scenario number three (perhaps the most common scenario): Your clinical load has been slowly growing. Almost without your realizing it, you now find yourself spending less and less time in the laboratory doing experiments until you are no longer doing benchwork at all. This alone would not necessarily be a problem. After all, many senior basic science faculty find themselves no longer doing actual benchwork. (Indeed, in a lab I worked at a long time ago, we’d go out of our way to dissuade our PI from venturing into the lab.) However, over time, your patient load continues to increase and you now find yourself spending less and less time even meeting with your lab personnel. You find that you are no longer even sure of what is going on in your lab on a day-to-day basis anymore. There are a pile of manuscripts that need to be finished, but you can’t get to them because you’re always in the clinic or in the O.R.. You try to work on them at home, but your wife and children demand their fair share of your attention when you manage to make it home for a while. You’ll soon be due to try to renew your grant, something you have no idea if you’ll be able to do now. You could try to cut back on your clinic time, but that would mean that patients’ waiting time to be seen would increase. Patients with cancer would be forced to wait longer. Also, if you cut back your clinical productivity, your department would not like it, because you would no longer be supporting your salary and overhead with clinical revenue, and this doesn’t even take into account the consideration that your referring physicians will not be pleased.
These are just a few examples, but in the end the conflicts all come down to the tension between two worlds which are very different, the world of the scientist and the world of the clinician. The world of the scientist values inquisitiveness and intellectually stimulation. It also tends to be less interested in practicality and more interested in intellectual pursuits and scientific novelty; i.e., answering questions that have never been asked or answered before. In constrast, the world of the clinician is almost purely practical. It tends to be task- and action-oriented, and protocol-driven. Asking and answering questions are valued, but only insofar as the questions and answers pertain to diagnosing and treating disease or overcoming problems that get in the way of good patient care. The clinician-scientist tries to bridge the gap between these two world in an increasingly difficult environment.
I hope I didn’t scare Tim off; so I’ll try to finish on a note of hope. Why do I continue to do it? I’ve often joked that, as an MD and a PhD, I catch equal crap from both worlds. Clinicians wonder if I know what I’m doing in the operating room, and basic scientists don’t think I can hack it as a basic scientist. So why put up with the stress of wearing two hats? Why not pick one or the other? There are two reasons. First, I think that clinician-scientists bring a unique perspective to the study of human disease that neither a clinician or a scientist alone can. Second, as I suspect must be the case with Tim, I want to make a difference. Nothing would be as satisfying as making a clinical observation, taking it to the laboratory, developing a treatment based on my laboratory observations, and then testing that in patients and seeing it work. True, I may never manage shepherd a treatment through all those stages (clinical observation, laboratory observations on the basic science, development of a therapy based on the science, and testing in clinical trials). After all, it can take a decade or more to do so, and I probably don’t have more than maybe two decades left in my career, but wouldn’t it be great if, before it’s time for me to retire, I manage to pull it off?

   The life cycle of translational research
David Gorski
Link
ResearchBlogging.orgI’m a translational researcher. To those of you who aren’t familiar with what that means, it means (I hope) that I study potential therapies in the lab and try to translate them into actual therapies that will cure patients of breast cancer — or, at the very least, improve their odds of survival or prolong survival when cure is not possible. Translational research is extremely important; indeed, it is the life blood of science-based medicine, with basic science producing the discoveries and clinical research the applications of these discoveries. When it works, it’s the way that science leads medicine to advance. However, sometimes I think that it’s a bit oversold. For one thing, it’s not easy, and it’s not always obvious what basic science findings can be translated into useful therapies, be it for cancer (my specialty) or any other disease. For another thing, it takes a long time. The problem is that the hype about how much we as a nation invest in translational research all too often leads to a not unreasonable expectation that there will be a rapid return on that investment. Such an expectation is often not realized, at least not as fast and frequently as we would like, and the reason has little to do with the quality of the science being funded. It has arguably more to do with how long it takes for a basic science observation to follow the long and winding road to producing a viable therapy. But how long is that long and winding road?
A lot longer than many, even many scientists, realize. At least, that’s the case if a paper from about a year ago by John Ioannidis in Science is any indication. The article appeared in the Policy Forum in the September 5 issue and is entitled Life Cycle of Translational Research for Medical Interventions. As you may recall, Dr. Ioannidis made a name for himself a couple of years ago by publishing a pair of articles provocatively entitled Contradicted and Initially Stronger Effects in Highly Cited Clinical Research and Why Most Published Research Findings Are False, which Steve Novella blogged about a couple of years ago.
Dr. Ioannidis lays it out right in the first paragraph:
Despite a major interest in translational research (1-3), development of new, effective medical interventions is difficult. Of 101 very promising claims of new discoveries with clear clinical potential that were made in major basic science journals between 1979 and 1983, only five resulted in interventions with licensed clinical use by 2003 and only one had extensive clinical use (4). Drug discovery faces major challenges (5-8). Moreover, for several interventions supported by high-profile clinical studies, subsequent evidence from larger and/or better studies contradicts their effectiveness or shows smaller benefits (9). The problem seems to be even greater for nonrandomized studies (9).
In order to figure out how long translational research can take to come to fruition, Dr. Ioannidis looked at some fairly high profile therapies (defined as therapies claimed to be effective in at least one study about it that has been cited 1,000 or more times) between 1990 to 2004. The reason that this particular definition was chosen is to use studies for which a milestone at which a therapy is widely recognized to be effective and safe. That, by the paper’s definition, is the point at which a basic science finding has finally been completely “translated” into an accepted therapy. The problem then becomes to identify exactly when such therapies began as a grain of a basic science finding or an idea based on basic science observations. This is not always easy to identify. In the case of one of my major areas of research interest, tumor angiogenesis, it’s relatively easy to cite Dr. Judah Folkman’s famous 1971 paper proposing the targeting of tumor angiogenesis as a therapy for cancer. Oh, we could quibble over whether that was truly the beginning. After all, it was hypothesized that angiogenesis was important to tumor growth decades before. However, the 1971 paper was the first explicit proposal to develop strategies to target tumor angiogenesis to treat cancer. Be that as it may, I’ve mentioned before that it was thirty years before an actual therapy used in humans was developed as an example of just how long it can take.
Dr. Ioannidis has tried to quantify this gap in a more general manner, which he refers to as the “translation lag.” What he found is quite sobering for people who are anxious for the rapid translation of basic science:
To place each discovery in time, we identified the year when the earliest journal publication on preparation, isolation, or synthesis appeared or the earliest patent was awarded (whichever occurred first). Overall, the median translation lag was 24 years (interquartile range, 14 to 44 years) between first description and earliest highly cited article (see the chart). This was longer on average (median 44 versus 17 years) for those interventions that were fully or partially “refuted” (contradicted or having initially stronger effects) than for nonrefuted ones (replicated or remaining unchallenged) (P = 0.004).
In a secondary analysis, we defined the time of discovery as the first description (publication or awarded patent) of any agent in the wider intervention class (those with similar characteristics and mode of action). Early translational work may be performed with different agents in the same class compared with those that eventually get translated into postulated high-profile clinical benefits. Analyses using information on the wider class of agents showed even longer translation lag, with median of 27 (interquartile range, 21 to 50) years and similar prolongations of the translation lag for refuted interventions.
This is represented in this table:
1298-1-med
One thing that you may notice on the table above that Dr. Ioannidis also did was that he also looked at some treatments or interventions that had been highly cited and were later refuted. What makes this interesting is that therapies that were never later refuted by clinical trials after the highly cited clinical trial that was used to support their use had a markedly shorter translation lag, 16.5 years (range 4 to 50 years) in the main analysis and 22 years (range 6 to 50 years) if the wider class of drugs is considered. In contrast, for remedies whose efficacy was later refuted, the translation lag was 44 years, as mentioned in the passage cited above. What could be the explanation? Dr. Ioannidis speculates:
We observed that most highly cited claims that were eventually refuted had a very slow translation history preceding them [e.g., flavonoids, vitamin E, and estrogens were already available for many decades before observational (nonrandomized) studies claimed implausibly large survival benefits in the 1990s]. We conclude that claims for large benefits from old interventions require extra caution as they are likely to be exaggerated. 
I’m half tempted to mention that there is a bit of an elephant in the room here in that it makes one wonder about complementary and alternative medicine, and how the existing studies would fare in this sort of analysis. Of course, one would be unlikely to find a paper on a CAM intervention cited 1,000 times showing efficacy. (In fact, for some it would be hard to find a paper strongly demonstrating efficacy of most CAM therapies.) But think of it this way: Many of these therapies have been around for hundreds of years, and there hasn’t been compelling evidence of their efficacy. Think of homeopathy, for instance. Samuel Hahnemann thought of it over 200 years ago, and we still don’t have evidence of its efficacy.
I know, I know, I’m dragging one of my personal peeves into this, and it is ridiculous to use the term “science” or “translational research” in referring to something like homeopathy, but I couldn’t resist. Mea culpa.
We can quibble about whether his methodology was the most appropriate or whether he picked the correct milestones to compare, but what Dr. Ioannidis shows is that, in essence, a lot of “translational” research takes close to two decades to bear fruit, and it’s fairly uncommon for it to take less than a decade. Moreover, as Dr. Ionnidis points out, less than 5% of promising claims based on basic science ever come to fruition as actual therapies. In other words, translational research is hard. Few promising ideas make it to therapies, and it takes a long time for those that do. Indeed, Dr. Ioannidis makes this recommendation:
Our analysis documents objectively show the long length of time that passes between discovery and translation. As scientists, we should convey to our funders and the public the immense difficulty of the scientific discovery process. Successful translation is demanding and takes a lot of effort and time even under the best circumstances; making unrealistic promises for quick discoveries and cures may damage the credibility of science in the eyes of the public.
His other recommendations are rather obvious: multidisciplinary collaborations with focused targets, incentives for testing claims in high quality, unbiased research, and the that large clinical trials with repeatability are required to demonstrate the efficacy of therapies. One recommendation, however, caught my eye:
New drug discovery is probably essential for common diseases where the existing drug armamentarium has been already extensively screened. Conversely, for uncommon and neglected diseases, the existing drug options may remain largely untested, and old drugs may find interesting new uses (12-14).
Actually, that’s a recommendation that’s been pretty obvious for a long time. For common diseases, existing drugs and related drugs have been tested and retested, and many, many variations on the old have been tried. For substantial breakthroughs, something new is required. Unfortunately, the profit incentive in drug manufacturing tends too much towards “me-too” drugs. The second part is actually very relevant to what I do. Testing such “off-label uses” of drugs can yield surprising results. Indeed, one of my two research projects is based on exactly such an off-label use. However, ironically it is a treatment for a very common disease, breast cancer, that I am testing; so Dr. Ioannidis is not entirely correct. Interesting new uses for drugs do not always make themselves known in uncommon and neglected diseases.
Regardless, however, what the public needs to understand is that translational research takes a long time. What they also need to understand is that the foundation upon which translational research rests is a strong core of basic science research. Without a wide variety of basic science findings upon which to base treatments, translational research has little to choose from in moving such findings to the next step. Contrary to popular belief as well, when there are robust basic science findings upon which to base a potential therapy, the road from laboratory bench discovery to usable therapy is long and winding.
REFERENCE:
D. G. Contopoulos-Ioannidis, G. A. Alexiou, T. C. Gouvias, J. P. A. Ioannidis (2008). Life Cycle of Translational Research for Medical Interventions Science, 321 (5894), 1298-1299 DOI: 10.1126/science.1160622

   Integrating patient experience into research and clinical medicine: Towards true “personalized medicine”
David Gorski
Link
We advocate science-based medicine (SBM) on this blog. However, from time to time, I feel it necessary to point out that science-based medicine is not the same thing as turning medicine into a science. Rather, we argue that what we do as clinicians should be based in science. This is not a distinction without a difference. If we were practicing pure science, we would be theoretically able to create algorithms and flowcharts telling us how to care for patients with any given condition, and we would never deviate from them. It is true that we do have algorithms and flowcharts suggesting guidelines for care for a wide variety of conditions, but there is wide latitude in them, and often a physician’s “judgment” still ends up trumping the guidelines. While it is also true that sometimes physicians have an overinflated view of the quality of their own “clinical judgment,” sometimes to the point of leading them to reject well-established science, as Dr. Jay Gordon frequently does, what I consider to be physician’s judgment is knowing how to apply existing medical science to individual patients based on their circumstances and, yes, even desires and values.
Indeed, if there’s one area where SBM has all too often fallen short in the past, it’s in taking into account the patient’s experience with various treatments. What got me thinking (again) about this issue was an article by Dr. Pauline Chen in the New York Times last Thursday entitled Listening to Patients Living With Illness. She begins her article with an anecdote:
Wiry, fair-haired and in his 60s, the patient had received a prostate cancer diagnosis a year earlier. When his doctors told him that surgery and radiation therapy were equally effective and that it was up to him to decide, he chose radiation with little hesitation.
But one afternoon a month after completing his treatment, the patient was shocked to see red urine collecting in the urinal. After his doctors performed a series of tests and bladder irrigations through a pencil-size catheter, he learned that the bleeding was a complication of the radiation treatment.
He recalled briefly hearing about this side effect three months earlier, but none of the reports he had been given or collected mentioned it, and once he had recovered from the angst of the emergency room and the doctor’s office visits and the discomfort of the clinical work-up, he didn’t give it more thought — until a few weeks later, when he started bleeding again.
By the time I met him, he was in the middle of his third visit to the hospital. “I feel like I’m tied to this place,” he said. He showed me a plastic jug partly filled with urine the color of fruit punch, and he described a post-treatment life marked by fear of going to the bathroom and discovering blood. “If I had known that my life would be like this after radiation,” he sighed, “I would have chosen the surgery.”
To this, I’ll add a little random bit of personal experience of my own. No, I wasn’t a patient who had to face something like this patient, but I do see something similar in my patients. Back when I was in my surgical oncology fellowship — and before that, in my general surgery fellowship — I was always taught that lumpectomy was preferable to mastectomy because it saves the breast and most women want to save their breasts. After all, lumpectomy plus radiation therapy results in the same chance of survival as mastectomy; so we should offer lumpectomy whenever tumor characteristics (the main one being size relative to the rest of the breast) permit it. Yet this assessment often neglects to acknowledge that, for some women, undergoing six or seven weeks of radiation is horribly inconvenient, and that there are often complications. It also often neglects to acknowledge that there is a price for saving the breast besides having to undergo radiation therapy: there’s the possibility of more surgery to achieve clear surgical margins, not to mention a higher risk of local recurrence in the breast. For some women, this latter possibility is a deal-breaker. Even though they acknowledge that their chances of survival would be the same with lumpectomy or mastectomy, the thought of an approximately 8% local recurrence rate eats at them to the point that they opt for mastectomy.
Then there is the issue of chemotherapy. We frequently recommend cytotoxic chemotherapy for women with relatively early stage breast cancer, even though the addition of chemotherapy in such patients only increases the chance of survival by perhaps 2–3% on an absolute basis, depending upon the tumor. Of course, as I’ve pointed out before, the benefits of chemotherapy are more marked in more advanced operable tumors, but in early stage tumors they are rather modest. This is therapy that causes hair loss, increased risk of infections, and can cause damage to the heart, but it is the standard of care. Most women are willing to undergo this sort of therapy, too; I can’t locate the study, but I’ve seen one survey where women respond that they would be willing to undergo chemotherapy for a 1% increased chance of survival.
The point is that these sorts of questions are value judgments that often depend upon what patients consider important. The patient described by Dr. Chen, for instance, would apparently prefer risks of surgery rather than peeing blood all the time and having to go back to the doctor’s office and hospital time and time again for this problem. Science can tell a physician and patient like this that radiation or surgery will produce an equivalent chance of surviving his cancer. It can tell them what the complications of each choice are likely to be, and what the odds are of each complication. That’s part of what I mean when I refer to science-based medicine. What it can’t tell the patient and doctor is which constellation of risks would be more easily bearable by the patient. The same is true for whether to choose mastectomy or radiation or whether to opt for chemotherapy after breast cancer. Science provides the numbers and the “price” of each choice, but it can’t — nor should it — tell the patient what to value. Moreover, what the patient values may not be what the physician values. As Dr. Chen points out:
Whether conducted at a laboratory bench or in clinical trials, medical research has long been driven by a single overriding goal — the need to find a cure. Usually referred to more modestly as a search for “the most effective treatment,” this standard has served as both a barometer of success and a major criterion for funding. Most published studies are marked by a preponderance of data documenting even minor blips in laboratory values or changes in the size of a spot of cancer or area of heart muscle damage on specialized X-rays. Some studies bolster the apparent success of their results with additional data on societal effects like treatment costs or numbers of workdays missed.
Few studies, however, focus on the patient experience.
She then refers to a study by Dr. Albert W. Wu, lead author and a general internist and professor of health policy and management at the Johns Hopkins Bloomberg School of Public Health in Baltimore published in the journal Health Affairs entitled Adding The Patient Perspective To Comparative Effectiveness Research. In this study, Wu et al argue for the inclusion of the patient’s perspective in comparative effectiveness research. What this involves is patient-reported outcomes. To illustrate the concept, Wu et al use this chart for patients with chronic obstructive pulmonary disease (COPD).
These sorts of measures are particularly appropriate for comparative effectiveness research (CER). For the reason, consider what CER is: basically CER compares existing treatment modalities already determined to be effective in prior clinical trials in order to determine which is more effective. Other important measures include cost-effectiveness. However, although some efforts go into assessing patient-reported quality of life outcomes of the sort listed above, all too often it’s hit-or-miss whether these sorts of measurements are included in clinical trials. One initiative that this article describes is the Patient-Centered Outcomes Research Institute, whose mandate is to:
    • Establish an objective research agenda;
    • Develop research methodological standards;
    • Contract with eligible entities to conduct the research;
    • Ensure transparency by requesting public input; and
    • Disseminate the results to patients and healthcare providers.
Wu et al suggest that the PCORI can only realize its potential if it supports initiatives that integrate measures of patient experience into not just research but into routine clinical care. A number of possibilities are suggested, including how to integrate general and disease-specific tools into clinical trials in order to measure patient-reported outcomes. Also suggested are various means of integrating these tools not just into clinical research but into routine clinical care, including using them in administrative claims data, linking this data to electronic medical records, and even promoting the collection of such data as being required for reimbursement.
One roblem I can perceive immediately in trying to use the PCORI is that it has no real power. In fact, the health insurance reform bill known as the Patient Protection and Affordable Care Act (PPACA), which mandated the creation of the Patient-Centered Outcomes Research Institute, provides no power to it. Indeed, its main charge is to assess “relative health outcomes, clinical effectiveness, and appropriateness” of different medical treatments, both by evaluating existing studies and conducting its own. Even given that huge mandate, the law also states that the PCORI does not have the power to mandate or even endorse coverage rules or reimbursement for any particular treatment. Indeed, so toothless is the PCORI, at least in its present form, that it has been disdainfully described as being like the UK’s NICE but without any teeth, which is all too true. Basically, the law says that Medicare may take the institute’s research into account when deciding what procedures it will cover, as long as the new research is not the sole justification and the agency allows for public input. Moreover, if the political reaction to the USPSTF’s revision of the guidelines for mammographic screening last year is any indication, if politicians don’t like a PCORI recommendation, you can be quite sure that they’ll behave similarly. After all the ranting about “rationing” that was used to attack the PPACA, it was not politically feasible to make the PCORI a government agency or to imbue it with any real authority.
Politics aside, let’s get back to the sorts of initiatives suggested by Wu et al. One that in particular interests me is the concept of using patient portals to collect this information. Patient portals are websites that offer a variety of services to patients, including secure e-mail communication with the clinician, the ability to schedule appointments and request prescription refills, as well as the opportunity to complete intake and other forms that used to be completed on paper in the office. The authors propose using such portals to collect patient-centered quality of life measurements and give an example of how this might be done in the case of a hypothetical breast cancer patient:
In one possible scenario, a woman with breast cancer is being followed by an oncologist who would like to know how she is doing on the chemotherapy regimen she is receiving. The oncologist logs on to PatientViewpoint.org, enters the patient’s number, and orders the BR-23 Breast Cancer–Specific Quality of Life Questionnaire for her to complete online before her next visit. The patient receives an e-mail notification to do this, logs on to PatientViewpoint.org, and completes the survey.
The patient’s results are automatically calculated and are made available both on the website and within the hospital’s electronic health record alongside all of her other laboratory test results. At the visit, the oncologist pulls up the results and asks the patient about an increase in her depression scores. It would also be possible to aggregate all of the patient’s questionnaire results with those of other patients receiving chemotherapy for similar breast cancer cases and to use these data to help compare the effectiveness of different regimens.
Dr. Wu’s site is currently only set up to accommodate breast and prostate cancer patients, but it could be expanded. There now exist a large number of tools like the BR-23 to assess quality of life, and, with what appears to be the nigh inevitable infiltration of the electronic medical record into medicine over the next several years, integrating such tools into routine clinical care should become increasingly easy and inexpensive. On the other hand, one problem with such tools is that clinicians are already buried in “information overload.” Whether they would actually read and use the results of such studies outside the context of clinical trials is not assured, at least not if there is no incentive to do so. If this sort of approach is going to work, the government and insurance companies are going to have to pony up. Another problem is that a lot of doctors don’t like this sort of measurement. They consider it unscientific and “squishy” or they don’t know what to do with the information. Whether these attitudes will change or not as CER becomes increasingly embedded in clinical research is impossible to say.
Dr. Wu’s article leads me to reflect upon two things. First, it’s important to remember that the reason these “softer,” “squishier” measures are becoming more important is precisely because SBM has been so successful. Diseases that were once fatal are now chronic. A prime example is HIV/AIDS. Back when I was in medical school, HIV was invariably fatal. AIDS patients died rapidly — and in most unpleasant ways. Thanks to SBM, which developed the cocktails of antiretroviral drugs, HIV/AIDS has become a chronic disease, so much so that babies born with HIV are now approaching adulthood. What this success means is that, although not completely, by and large mortality is no longer the be-all and end-all of HIV treatment. Now, we are seeing quality of life issues coming to the fore. The same is true for some cancers, and it’s certainly true for diabetes and heart disease. As Wu et al point out:
Patient-reported outcomes directly support the primary goal of much of health care: to improve health-related quality of life, particularly for people with chronic illnesses. No one can judge this better than the patient. For example, the main objective of hip replacement surgery is to reduce pain and improve the capacity to get around. The main goal of cataract extraction is to improve visual functioning—that is, the ability to perform activities that require eyesight, such as reading, walking without falls, and working on a computer.
In addition, there are often trade-offs between the length and quality of life. Important considerations are the side effects of treatment of HIV disease, the temporary diminution of functioning after coronary bypass surgery, or fatigue resulting from cancer chemotherapy. Even for life-saving treatments, this kind of trade-off can influence a patient’s decision making among alternative courses of care.
Once again, these decisions and the trade-offs patients decide to accept should be informed by the science. The options presented to the patient and their cost in terms of potential complications and impact on the patient’s ability to go about his daily activities and in essence live his life must be based on science. However, that does not mean that the final determination will always be purely based on estimates of efficacy. If the patient decides, for instance, that the survival advantage that chemotherapy will provide after her breast cancer surgery is not sufficient to be worth months of hair loss, fatigue, and the risk of heart damage, then that is her choice. The key is that we as clinicians must make sure that she has accurate, science-based information upon which to base that choice. Informed consent must be based on sound, scientifically verified information. Anything else, such as the sorts of “informed consent” advocated by “health freedom” groups is in reality misinformed consent. It is our responsibility as science-based practitioners to do our best to make sure that the treatments we offer our patients are based in science and that the information about the relative benefits, risks, and costs about these treatments is also based in science.
The second thing that comes to my mind is the complete contrast between the sorts of efforts that Wu et al are undertaking and what purveyors of unscientific so-called “complementary and alternative medicine” (CAM) do. SBM is, through CER, undertaking systematic measurements of quality of life measures, and the use of genetic tests that provide information about prognosis and predict response to therapy, making its first real steps towards truly “personalized” medicine. Yes, these steps are halting — stumbling at times, even — but they are steps towards the day when SBM can offer patients treatment options based on science and personalized to the characteristics of the biology of their disease that are unique to them, all while taking the patients’ own values and desires into account. Contrast that to so-called CAM, where “personalized medicine” basically means making it up as the practitioner goes along, and I think you’ll see what I mean. Whatever the deficiencies and faults of SBM (and it’s impossible not to concede that there are many), SBM is far closer to true “personalized medicine” than any CAM, and it is using CER to come even closer still. CAM has nothing to compare.

Testing the “individualization” of CAM treatments
Brennen McKenzie on July 6, 2012
Link
One of the common claims of alternative medicine practitioners is that they individualize their treatment while conventional medicine treats all patients the same. This is nonsense on several levels, but it is also a common excuse for why randomized clinical trials cannot be performed, or cannot be viewed as reliable evidence, in evaluating some alternative therapies. However, some trials have been done that attempt to account for this supposed individualization of therapy, and generally they have failed to show a benefit to the supposedly individualized approach. One of those, involving Traditional Chinese Medicine (TCM) was recently discussed by Edzard Ernst, one of few, and most productive researchers in the CAM field applying an evidence-based approach:
Matthias Lechner, MD, Iva Steirer, MD, Benno Brinkhaus, MD, Yun Chen, CMD, Claudia Krist-Dungl, MS, Alexandra Koschier, MS, Martina Gantschacher, MA, Kurt Neumann, MS, and Andrea Zauner-Dungl, MD. Efficacy of Individualized Chinese Herbal Medication in Osteoarthrosis of Hip and Knee: A Double-Blind,Randomized-Controlled Clinical Study. The Journal of Alternative and Complementary Medicine. 2011;17(6): 539–547.
First, why is the notion that CAM is somehow more individualized than conventional care total nonsense? Well, to begin with, any good doctor considers the particular history, physical examination findings, diagnostic test results, known medical problems, and concurrent therapies of each patient. If individualized treatment simply means considering the unique circumstances and values of the particular patient you are treating, then all good medicine is individualized. That concept is even built into the common definitions of evidence-based medicine:
Evidence based medicine is the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients. The practice of evidence based medicine means integrating individual clinical expertise with the best available external clinical evidence from systematic research.
(Sackett)
[EBM is] the integration of the best research evidence with our clinical expertise and our patient’s unique values and circumstances.
(Strauss)
However, when CAM practitioners claim that formal scientific research and science-based medicine ignore individual variation, they are usually referring to the practice of studying groups of patients under controlled conditions and then applying lessons learned from those studies to the care of individuals. They claim that since we are all snowflakes, utterly unique, what is learned from groups cannot tell us anything useful about individuals.
This argument fails most dramatically on the simple evidence of the tremendous effectiveness of science-based medicine. Tens of thousands of years of looking at patients one by one and trying to figure out based on those experiences what to do for the next patient failed to control or eliminate any common diseases or meaningfully improve the length and quality of human life and health. A couple of centuries of gradually relying on formal scientific research instead of such haphazard individual experiences has wiped out or dramatically reduced many common and deadly diseases, nearly doubled average life expectancy (at least for those who can afford to use science-based medicine), and in many other ways unequivocally improved our health. It requires deep self-delusion to deny that science works better than prescientific, unstructured ways of figuring out how to preserve and restore health.
On a more theoretical level, however, consider this. Statistics can indicate the probability of winning or losing a game of chance very precisely, on the group level. As an individual, of course, you can’t know with certainty whether you will win or lose if you go to Las Vegas and play these games because these statistics only describe what happens over the course of many trials, that is what will happen on average when large numbers of people play. They don’t predict for you, as a unique individual, what your results at blackjack or roulette will be. This is very much like the situation in science, where controlled studies look at outcomes on the level of the group but can’t precisely predict the results of a therapy in an individual patient.
And yet, casinos make enormous sums of money by playing the odds and expecting that most people will lose. This is a successful strategy for them. And many people lose, some with disastrous personal consequences, by imagining that they are exempt from the statistical rules that apply to groups and that some special individual factor will allow them to beat the odds. Choosing to believe that general statistical principles don’t apply to them because they are special and unique ruins people’s lives in Vegas, and in medicine. Choosing to play with or against the odds, as defined by formal research, is no guarantee, but it is much more likely to lead to a good outcome than imagining the odds don’t matter because each of us is unique.
Finally, when a CAM practitioner claims they treat every individual patient based on that patient’s unique characteristics and that science-based medicine treats all patients as if they were the same because it bases treatment guidelines on research done on groups, they are simply mistaken. If a series of controlled studies indicates that Treatment X is better than Treatment Y for a certain diseases, and if I use this to support giving patients with that disease Treatment X most of the time, then yes I am applying information gained from population research to individuals. I am playing the odds.
However, if a CAM practitioner looks at a patient and evaluates their particular characteristics and then decides on a specific treatment, where do they come up with the connection between the patient’s characteristics and the treatment? They use their personal experience, gained from seeing what happens with prior patients, or they use rules laid down by other practitioners based of their own experiences, or they rely on general rules based on the theoretical ideas behind the style of therapy they use. In other words, they extrapolate from observations made on other patients to the individual they are currently treating.
This is exactly the same as what a science-based practitioner does with one important difference: the generalizations that scientific medicine applies to individual patients come from formal, controlled research designed to compensate for the unreliability of individual observations and judgments, whereas the generalizations used by CAM come from informal, unstructured observations with no control for bias or the many common errors that mislead us when we study disease. CAM practitioners are using generalizations based on the study of groups to decide how to treat each new case, they are just relying on poorer quality group evidence.
Ok, so how does this apply to the clinical study that looked at supposedly individualized TCM herbal therapy for arthritis of the knee? Well, the study started by randomly assigning patients to either receive individualized herbal treatment based on the judgment of experienced practitioners in each case, or a standard herbal mixture believed, again based on past experience with groups of patients, not to have any benefits for arthritis. The experimental formula consisted of a number of herbs selected in advanced based on TCM theory which the investigators expected might be useful for the kind of disease they were studying. However, individual patients received particular combinations of these ingredients based on the judgment or practitioners at the time they were evaluated.
The control treatment (not really a placebo since it contained chemicals which might or might not have real physiological effects, since none have been thoroughly evaluated in scientific studies) was a collection of herbs not believed to have benefit for arthritis based on prior experience and TCM theory It was made to taste similar to the herbs pre-selected for the experimental treatment to help make it harder for patients to know which they were receiving. I can already hear the complaints of some herbalists that this makes it an inappropriate control since taste is one of the guiding principles for the use of herbs in some approaches to herbal medicine. I’ll leave that pseudoscientific objection aside for now since it’s not directly relevant to the point here.
Baseline characteristics were similar between the two groups of patients, and randomization and blinding appeared to be properly conducted. Overall the study was well-done methodologically, with a formal accounting of patients lost to follow-up and a reasonable effort to use standard and predefined outcome measures.
So what were the results? Well, as is usual in a study looking at a subjective measure like pain, all patients improved. There was, however, no difference between those who received individualized treatment and a random herbal concoction not expected to have any effect on arthritis. This most likely indicates nothing happening here other than nonspecific effects associated with participating in a trial, including placebo, regression to the mean, the Hawthorne effect, and all the usual suspects that fool us in clinical trials, and in real life.
This study nicely illustrates several of the issues associated with supposed individualization of CAM treatment. First, it shows that such treatment is not, in any meaningful sense, any more individualized than good quality science-based medical treatment. Choosing a selection of herbs based on previous experience, historical use, tradition, and the unscientific theories of Traditional Chinese Medicine, and then selecting which of these herbs to give each patient based on the same prior experience and unscientific theory, is still applying generalizations based on groups to individuals. It simply uses generalizations based on unreliable sources of data.
The study also illustrates that individualizing therapy in this way doesn’t add any efficacy to the treatment. Not surprisingly, the study showed, as the others mentioned early have as well, that tailoring treatment to individuals based on generalizations derived from biased and unreliable sources of information leads to a therapy no more effective than randomly picking herbs out of a hat.
The difference between effective science-based medicine and ineffective medicine of any kind, conventional or alternative, is that the general principles used to guide therapy are derived from formal, controlled research that compensates for the weaknesses in our individual, informal, and unstructured judgment. If individualized medicine is just a code for using informal group observations instead of structured scientific ones to guide therapy, than it is not surprising that it doesn’t work any better than just making up a treatment haphazardly with no guiding principles at all.

   “Motivated reasoning,” alternative medicine, and the anti-vaccine movement
David Gorski 
Link
One theme that we at Science-Based Medicine keep revisiting again and again is not so much a question of the science behind medical therapies (although we do discuss that issue arguably more than any other) but rather a question of why. Why is it that so many people cling so tenaciously to pseudoscience, quackery, and, frequently, conspiracy theories used by believers to justify why various pseudoscience and quackery are rejected by mainstream science and medicine? Certainly, I’ve touched on this issue before on several occasions, for example, with respect to the anti-vaccine movement, the claim that abortion causes breast cancer, and how we as humans crave certainty.
It turns out that science and science-based medicine are hard for humans to accept because they often conflict with what our senses perceive and brains interpret as irrefutable evidence. The pattern-seeking function of our brain, when evaluating questions of causation in medicine, frequently betrays us. For instance, when a parent sees her child regress into autism sometime not long after being vaccinated, the easiest, most instinctive, and most emotionally compelling conclusion is that the vaccine must have had something to do with it. When scientists tell her that, no, in large studies looking at hundreds of thousands of children, there is no good evidence that vaccination confers an increased risk of autism and a lot of evidence that it does not, it’s a very hard message to believe, because it goes against how the parent interprets what she’s seen with her own eyes. Indeed, how often have we seen believers in the vaccine-autism link pour derision on the concept that when something like autistic regression happens in close temporal proximity to vaccination that the correlation does not necessarily equal causation? Similarly, believers in “alternative medicine” who experience improvement in their symptoms also pour derision on the observation, explained so well by R. Barker Bausell in Snake Oil Science, that people frequently take remedies when their symptoms are at their worst, leading them to attribute natural regression to the mean to whatever nostrum they started taking at the time.
These issues have come to the fore again, thanks to an article by an acquaintance of mine, Chris Mooney, author of The Republican War on Science, Storm World: Hurricanes, Politics, and the Battle Over Global Warming, and Unscientific America: How Scientific Illiteracy Threatens our Future (co-authored with Sheril Kirshenbaum). The article appeared in a recent issue of Mother Jones and was entitled, rather ironically, The Science of Why We Don’t Believe Science. Chris made his name as an author primarily in writing about the science of anthropogenic global warming and the political battles over policies intended to mitigate it and, to a lesser extent, over creationism and evolution denial. Of late he has written about the anti-vaccine movement as an anti-science movement, leading predictably to his being attacked by the likes of J.B. Handley as viciously as I and others have. Also of note, although he was widely praised for The Republican War on Science and Storm World, Mooney has been widely criticized in some circles for being too critical of “new atheists” and for lack of substance. In his current article, he discusses some of the science thus far about why people can cling to beliefs that science doesn’t just cast doubt upon but shows convincingly are totally wrong.
Motivated reasoning
In his article, Mooney sets the stage with a very famous example studied by Stanford University psychologist Leon Festinger in the 1050s of the Seekers. The Seekers were an apocalyptic cult in the Chicago area led by a Dianetics enthusiast named Dorothy Martin. Its members believed that they were communicating with aliens, one of whom was named “Sananda,” who was supposedly the astral incarnation of Jesus Christ. Martin also taught her followers that Sananda had told her the precise date of a world-ending cataclysm: December 21, 1954. As a result, some of Martin’s followers quit their jobs and sold their homes because they expected that a spaceship would rescue them right before the earth split open and the sea swallowed much of the United states. In fact, Martin’s followers even went so far as to rid themselves of all traces of metal, even removing underwire bras and taking the zippers out of their clothes, because they were told that such metal would pose a danger to the spaceships. Here’s Mooney’s account of what happened when December 21, 1954 came and went and, as those of us living today know, no cataclysm occurred:
At first, the group struggled for an explanation. But then rationalization set in. A new message arrived, announcing that they’d all been spared at the last minute. Festinger summarized the extraterrestrials’ new pronouncement: “The little group, sitting all night long, had spread so much light that God had saved the world from destruction.” Their willingness to believe in the prophecy had saved Earth from the prophecy!
From that day forward, the Seekers, previously shy of the press and indifferent toward evangelizing, began to proselytize. “Their sense of urgency was enormous,” wrote Festinger. The devastation of all they had believed had made them even more certain of their beliefs.
In the annals of denial, it doesn’t get much more extreme than the Seekers. They lost their jobs, the press mocked them, and there were efforts to keep them away from impressionable young minds. But while Martin’s space cult might lie at on the far end of the spectrum of human self-delusion, there’s plenty to go around. And since Festinger’s day, an array of new discoveries in psychology and neuroscience has further demonstrated how our preexisting beliefs, far more than any new facts, can skew our thoughts and even color what we consider our most dispassionate and logical conclusions. This tendency toward so-called “motivated reasoning” helps explain why we find groups so polarized over matters where the evidence is so unequivocal: climate change, vaccines, “death panels,” the birthplace and religion of the president (PDF), and much else. It would seem that expecting people to be convinced by the facts flies in the face of, you know, the facts.
I’ve actually written about motivated reasoning before a couple of years ago. At the time, I used a then-recent study that examined how impervious to evidence certain beliefs about politics were, specifically the belief that Saddam Hussein had been involved in planning 9/11, conspiring with Al Qaeda to destroy the World Trade Center twin towers. In this study, even President George W. Bush’s own words stating that Hussein was not involved in planning 9/11 were not enough to convince believers. Another study cited used similar methodology regarding Saddam Hussein’s lack of weapons of mass destruction. In fact, in this study, there was a “backfire” effect, in which those exposed to disconfirmatory information about Saddam Hussein’s involvement with 9/11 were actually more likely to believe that he was, in fact, involved. Also discussed was the belief that President Barack Obama was not born in the United States and is therefore not eligible to be President (the “Birther” movement, which recently suffered a bit of a setback) and the belief that there were “death panels” written into the recently passed Patient Protection and Affordable Care Act. In the study I discussed, the authors based their analysis of motivated reasoning on its being driven primarily by cognitive dissonance, the the feeling we have when we are forced to become aware that we are holding two contradictory thoughts at the same time. The strength of the dissonance depends upon the importance of the subject to an individual, how sharply the dissonant thoughts conflict, and how much the conflict can be rationalized away, and cognitive dissonance theory thus posits that, when faced with evidence or occurrences that challenge their beliefs, people will tend to minimize the dissonance any way they can without giving up those beliefs.
To the list of examples provided by the authors, I also added the example of someone well-known to this blog, namely Andrew Wakefield, the (in)famous British gastroenterologist who in 1998 published a study in The Lancet that claimed to find a link between the MMR vaccine and “autistic enterocolitis.” When revelations of Wakefield’s financial fraud came to light, however, his fans in the anti-vaccine movement were motivated to cling all the more tightly to him, circling the wagons and attacking anyone who had the temerity to point out his fraud, bad science, bad medicine, and massive conflicts of interest. For example, just last month, in response to criticism of Andrew Wakefield, J.B. Handley, the founder of the anti-vaccine group Generation Rescue, pointed out that people like him view Andrew Wakefield as “Nelson Mandela and Jesus Christ rolled up into one.” Never mind that, scientifically speaking, Wakefield is just as discredited in his science as Dorothy Martin was in her predictions of global destruction. In the same article, anti-vaccine activist Michelle Guppy warned the reporter direly, “Be nice to him, or we will hurt you.” As you can see, despite the drip, drip, drip of allegations and evidence showing Andrew Wakefield to be a horrible scientist and even a research fraud have not had much of an effect on committed activists. I would argue, however, that they did have a significant effect on the media and the fence-sitters.
For the most part, most scientifically literate people know what cognitive dissonance is, but what is “motivated reasoning”? According to Mooney, to understand motivated reasoning, you first have to understand that what we humans call “reasoning” is not a cold, emotionless, Mr. Spock-like process. The way we human beings reason is actually suffused with emotion, or affect:
Not only are the two inseparable, but our positive or negative feelings about people, things, and ideas arise much more rapidly than our conscious thoughts, in a matter of milliseconds—fast enough to detect with an EEG device, but long before we’re aware of it. That shouldn’t be surprising: Evolution required us to react very quickly to stimuli in our environment. It’s a “basic human survival skill,” explains political scientist Arthur Lupia of the University of Michigan. We push threatening information away; we pull friendly information close. We apply fight-or-flight reflexes not only to predators, but to data itself.
We’re not driven only by emotions, of course—we also reason, deliberate. But reasoning comes later, works slower—and even then, it doesn’t take place in an emotional vacuum. Rather, our quick-fire emotions can set us on a course of thinking that’s highly biased, especially on topics we care a great deal about.
As a result, if this hypothesis is accurate, it can be expected that people will almost always respond to scientific or technical evidence in a way that justifies their preexisting beliefs. Examples of evidence that support this hypothesis are listed, including the study I discussed two years ago using the example of the persistent belief that Saddam Hussein had a hand in engineering 9/11. Also discussed was a classic study from 1979 in which pro- and anti-death penalty advocates were exposed to two fake studies, one supporting and one refuting the hypothesis that the death penalty deters violent crime. In addition, they were also shown detailed scientific critiques of each study that indicated that neither study was methodologically stronger than the other. In each case, advocates were more likely to find the study that supported their bias more convincing and to be more critical of the one that did not.
To anyone who understands human nature, this is not particularly surprising. After all, as Simon & Garfunkel sang in their 1970 song The Boxer (one of my all time favorite songs), “a man hears what he wants to hear and disregards the rest.” That’s not quite motivated reasoning, but close. Motivated reasoning would be more along the lines of saying, “a man pays attention to information that supports his beliefs and values and finds ways to disregard or discount the rest.” This principle, more than anything else, probably explains why believers in alt-med and anti-vaccine activists are immune to disconfirming evidence. Not just immune, either, they actively seek out confirming evidence and avoid disconfirming evidence, a task made much easier by the Internet and multiple different news outlets catering to different ideologies:
Okay, so people gravitate toward information that confirms what they believe, and they select sources that deliver it. Same as it ever was, right? Maybe, but the problem is arguably growing more acute, given the way we now consume information—through the Facebook links of friends, or tweets that lack nuance or context, or “narrowcast” and often highly ideological media that have relatively small, like-minded audiences. Those basic human survival skills of ours, says Michigan’s Arthur Lupia, are “not well-adapted to our information age.”
We see this in the CAM movement. An entire network of websites and blogs has sprouted up over the last decade or so. CAM believers, if they wish, can peruse sites like NaturalNews.com, Mercola.com, and Whale.to, watch television shows like The Dr. Oz Show, and never see a single piece of information or study that challenges their world view that because it’s natural it must be better, that conventional, scientific medicine is hopelessly in the thrall of big pharma, and that modalities that are nothing more than magical thinking can cure disease. Similarly, anti-vaccine activists have their own set of websites, including Generation Rescue, Age of Autism, the NVIC, the Orwellian-named International Medical Council on Vaccination (formerly “Medical Voices,” and discussed by Mark Crislip and myself), SafeMinds, and many others. These CAM and anti-vaccine sites also have their own scientific-seeming meetings, such as Autism One (which, by the way, is fast approaching again) and the AANP.
Wrapped safely in such a cocoon, believers seldom encounter arguments against their cherished beliefs, much less strong arguments against them. No wonder they’re often so poor at defending their favorite woo when they dare to stray out of the safe confines of their little world. However, one interpretation of motivated reasoning that I’ve come up with states that you don’t actually have to be good at producing arguments that convince other people; you just have to be good enough to cherry pick arguments that convince yourself.
Politics, CAM, and the anti-vaccine movement
While Mooney’s summary for the evidence for motivated reasoning is compelling, he stumbles a bit in trying to ascribe different forms of motivated reasoning to the right and the left. While it is clear that certain forms of anti-science do tend to cluster either on the right or the left (for example, anthropogenic global warming denialism is definitely far more common on the right), if motivated reasoning is a valid hypothesis that describes well how human beings react to information that challenges their belief systems and values matter more (at least initially) than facts and science, then it would only be expected that certain forms of science would be viewed more hostilely by the right than the left while other scientific findings would be viewed more hostilely by the left. Unfortunately, one of the examples Mooney picks is fairly dubious:
So is there a case study of science denial that largely occupies the political left? Yes: the claim that childhood vaccines are causing an epidemic of autism. Its most famous proponents are an environmentalist (Robert F. Kennedy Jr.) and numerous Hollywood celebrities (most notably Jenny McCarthy and Jim Carrey). The Huffington Post gives a very large megaphone to denialists. And Seth Mnookin, author of the new book The Panic Virus, notes that if you want to find vaccine deniers, all you need to do is go hang out at Whole Foods.
It’s hard not to note right here that the founder of Whole Foods, John Mackey, is an anti-union Libertarian and admirer of Ayn Rand. In any case, I really hate it when people like Mooney try to pin anti-vaccine views as being mainly “on the left.” True, left-leaning crunchy types are the primary face of anti-vaccine views, but there is an entire underground on the right that is virulently anti-vaccine. These include General Bert Stubblebine III‘s Natural Solutions Foundation, far right libertarians, and others who want to protect their “purity of essence.” In addition, FOX News isn’t above pushing anti-vaccine nonsense. For example, of late the FOX and Friends crew has been doing sympathetic pieces on Andrew Wakefield, interviews with Dr. Bob Sears, SafeMinds’ anti-vaccine PSA campaign, Louise Kuo Habakus (who is virulently anti-vaccine herself and politically active in New Jersey pushing for transparent “philosophical exemption” laws. Politically, some of the most rabid anti-vaccine activists in government are conservative, for instance Representative Dan Burton. Moreover, conservative fundamentalist religion is not uncommonly a motivation for anti-vaccine views. Not surprisingly, Mooney’s example ignited a rather intense debate in the blogosophere, which included Mike the Mad Biologist, Razib Khan, Joshua Rosenau, Andrew Sullivan, David Frum, and Kevin Drum, among others.
This debate didn’t go very far in either direction because there aren’t actually a lot of good data examining whether there is a correlation between political affiliation and anti-vaccine views. Ultimately, Mooney followed up with a post on his blog in which he did the best he could do with polling data on the politics of vaccine resistance. Reanalyzing a poll from 2009 asking about Jenny McCarthy’s anti-vaccine views, specifically how many people were aware of them and how many were more or less likely to agree with them, Brendan Nyhan and Chris Mooney found:
So here are the results: Liberals (41% not aware, 38 % aware but not more likely, 21 % aware and more likely); Moderates (48% not aware, 28% aware but not more likely, 24% aware and more likely); Conservatives (49% not aware, 28 % aware but not more likely, 23% aware and more likely).
These results basically suggest that there’s little or no political divide in terms of who falls for Jenny McCarthy’s misinformation. Notably, liberals were somewhat more aware of her claims and yet, nevertheless, were least likely to listen to them. But not by a huge margin or anything.
Mooney also noted another poll done by Pew regarding whether vaccines should be mandatory:
What’s interesting here is that Pew also provided a political breakdown of the results, and there was simply no difference between Democrats and Republicans. 71% of members of both parties said childhood vaccinations should be required, while 26% of Republicans and 27% of Democrats said parents should decide. (Independents were slightly worse: 67% said vaccinations should be required, while 30% favored parental choice.)
Bottom line: There’s no evidence here to suggest that vaccine denial (and specifically, believing that childhood vaccines cause autism) is a distinctly left wing or liberal phenomenon. However, I will reiterate that we don’t really have good surveys at this point that are clearly designed to get at this question.
Even though the evidence is admittedly weak and more studies and surveys would definitely be in order, Mooney’s conclusion is nonetheless in line with my experience. I’ve said before many times that anti-vaccine views are the woo that knows no political boundaries. Although I don’t have hard scientific data to support this my contention and therefore can’t definitively discount the possibility that my observations represent confirmation bias, I’ve noticed that right wing anti-vaccine activists tent to be suspicious of the government and appeal to “health freedom” as a reason for their resistance to vaccination, and tend to eschew any societal obligation to contribute to herd immunity. Left wing anti-vaccine activists tend to be suspicious of big pharma and believe that vaccines are somehow “unnatural.” I realize my interpretation might be biased, but until better data are available it’s all I have to work with. Similarly, alternative medicine use tends not to fall into an easy left-right dichotomy either. My favorite example to illustrate this point is that, even though alternative medicine is viewed as a crunchy, “New Age” phenomenon more prevalent on the left, the Nazi regime actively promoted naturopathy and various other “volkish” alternative medicine modalities. I trust that now someone will invoke Godwin’s law, but forgive me; I was intentionally using an extreme example to illustrate my point that all parts of the political spectrum can be prone to quackery.
Finally, Mooney makes another point that I quibble with:
Well, according to Charles Taber and Milton Lodge of Stony Brook, one insidious aspect of motivated reasoning is that political sophisticates are prone to be more biased than those who know less about the issues. “People who have a dislike of some policy—for example, abortion—if they’re unsophisticated they can just reject it out of hand,” says Lodge. “But if they’re sophisticated, they can go one step further and start coming up with counterarguments.” These individuals are just as emotionally driven and biased as the rest of us, but they’re able to generate more and better reasons to explain why they’re right—and so their minds become harder to change.
I would quibble somewhat with whether, in the case of science and medicine at least, that apparent “sophisticated” understanding of the issues possessed by ideologues is actually as sophisticated as it appears on the surface. In some cases it might be, but far more often it’s a superficial understanding that has little depth, mainly because few lay people have the detailed scientific and medical background to apply the information. It’s often a matter of knowing facts, but not having the scientific experience, understanding of mechanisms, or sophistication to put them in context or to apply them to the situation properly. Thus, the arguments of, for instance, anti-vaccine advocates often have the veneer of scientific sophistication, but to those knowledgeable about vaccines are easily identified as utter poppycock. Examples abound, and include this “review” article by a man named David Thrower is, and every “scientific review” published by, for example, Age of Autism.
I can’t remember how many times that, while “debating” in misc.health.alternative, I would have a study quoted to me as supporting an antivaccination or other alternative medicine viewpoint and find that, when I actually took the trouble to look up the study and download the PDF of the actual article rather than just reading the abstract (which is all most lay people have access to and therefore all they read), I would find a far more nuanced and reasonable point or even that the article didn’t support what the altie was saying. One other aspect that often comes into play is an extreme distrust of conventional medicine and/or the government such that few individual studies that question the safety of vaccines are given far more weight in their minds than the many more studies that show vaccines to be extraordinarily safe or large metanalyses. Certainly this is one reason why the infamous Wakefield study, despite being shoddily designed and now thoroughly discredited, keeps rearing its ugly head again and again and continues to be cited by antivaccination activists as strong evidence that the MMR vaccine causes autism. Basically, what is happening here is that highly intelligent and motivated people can construct arguments that seem better to the uninformed.
One thing that must be remembered about motivated reasoning is that we as skeptics and supporters of science-based medicine must remember that, as human beings, we are by no means immune to this effect. Indeed, as Mooney points out, citing recent research, it’s quite possible that reasoning is a better tool for winning arguments than it is for finding the truth, and when motivated reasoning combine with the echo chamber effect of modern social groups bound together by the Internet and like-minded media, the result can be disastrous for science:
But individuals–or, groups that are very like minded–may go off the rails when using reasoning. The confirmation bias, which makes us so good at seeing evidence to support our views, also leads us to ignore contrary evidence. Motivated reasoning, which lets us quickly pull together the arguments and views that support what we already believe, makes us impervious to changing our minds. And groups where everyone agrees are known to become more extreme in their views after “deliberating”–this is the problem with much of the blogosphere.
Actually, I’m constantly asking myself when I’m writing one of these logorrheic gems of analytic brilliance if I really am being analytically brilliant or am I being selectively analytically brilliant in order to bolster my pre-existing beliefs and values? In other words, am I doing from the other viewpoint the same things that anti-vaccine zealots, for example, do when they cherry pick and misrepresent studies in order to support their beliefs that vaccines cause autism? Of course, that’s where our readers come in, as does the fact that I (and, I have no doubt, every other SBM blogger) frequently ask myself that very question. As Richard Feynman famously said, “The first principle is that you must not fool yourself – and you are the easiest person to fool.” Science is simply a method for minimizing the chance that you will fool yourself. To say “I saw it with my own eyes” is not enough, but that is what our brains are hard-wired to believe.
That’s one reason why I’m far less concerned about winning over committed ideologues. Although such a task is possible and people do change their minds, sometimes even about things very important to them, for the most part expecting to win over someone like J.B. Handley, Jenny McCarthy, or Barbara Loe Fisher is a fool’s errand. The people who need to be educated are the ones who are either on the fence or otherwise susceptible to pseudoskeptical, sophisticated-sounding arguments from denialists because they do not understand science or the issues. Although it will by no means be easy, such a goal is at least achievable.

   Using attacks on science by the anti-vaccine movement as a “teachable moment”
David Gorski
Link
Last week, I wrote one of my usual ridiculously detailed posts analyzing a recent study (Price et al) that, if science and reason ruled, would be the last nail in the coffin of the hypothesis connecting autism with the mercury-containing preservative, thimerosal, which used to be in many childhood vaccines but was phased out beginning in 1999 and disappearing in infant vaccines except for the flu vaccine by early 2002. Of course, for at least the last five years, the thimerosal-autism hypothesis has been a notion whose coffin already had so many nails pounded into it that Price et al probably had a hard time finding even a tiny area of virgin wood into which to pound even a tiny nail of a study published in an impact factor one journal, much less the spike that their study in Pediatrics represented.
Unfortunately, as we know, in the anti-vaccine movement unreason rules, and, not unexpectedly, as a result this study has changed little in the debate, the fortuitously ironic happenstance of its being released the day before Mark Blaxill and Dan Olmsted’s anti-mercury screed Age of Autism not withstanding. To physicians and scientists, it is another strong piece of data being added to the confluence of evidence that has shown no link between mercury in vaccines and autism (or vaccines themselves and autism, for that matter). It is yet another confirmation that vaccines are safe. In contrast, to the anti-vaccine movement, it is simply yet another confirmation that the CDC is hopelessly biased, that scientists are in on a conspiracy to suppress The Truth, and that they are the poor persecuted minority, the only ones who know What Is Really Going On.
When I wrote my post last week, I didn’t know whether or not it would be worth my while to comment on the response of anti-vaccine activists to the study. The reason is that, as fun as it is to reveal their responses to be as vacuous as they are, I wasn’t sure that it would be educational. Granted, sometimes educational value takes a back seat to criticism, but sometimes it’s just too easy. In any case, by mid-week, there had been virtually no criticism of the study yet from the usual sources; so I figured it to be a moot point whether or not I would end up writing about this study one last time. Then, on Thursday morning I noted an e-mail in my in box. In order to keep my finger on the pulse of various pseudoscience movements, I subscribe to e-mail lists of various crank organizations, one of which is Generation Rescue and another of which is SafeMinds. SafeMinds, as you may recall, is the organization headed up by Sallie Bernard. As you may also recall, Bernard was originally on the external consulting committee that participated in the design of Price et al, and, before it, Thompson et al, the two of which ultimately made up a one-two punch against the mercury-autism hypothesis. When she saw that the results of Thompson et al were going against her idea and that no link between thimerosal-containing vaccines and neurodevelopmental disorders was showing up in the preliminary analyses, she resigned from the committee and started attacking Thompson et al. What surprised me was that she wasn’t ready with a criticism of Price et al when it was released.
OF PHARMA SHILLS AND IGNORANCE OF EPIDEMIOLOGICAL STUDY DESIGN
Queue my e-mail, wherein various crank organizations happily send me blogging material. What was sitting in it on Thursday morning was a missive from SafeMinds that linked to what SafeMinds apparently thought to be a rebuttal of Price et al. Then, right on the website of what I consider to be currently the ultimate crank anti-vaccine propaganda blog, I saw the headline SafeMinds Response to Thimerosal and Autism Pediatrics Study. When I read it, I realized that I had a “teachable moment,” because SafeMinds’ response to the study demonstrated such a misunderstanding of basic clinical trial design that I thought I could use it to teach our readers a bit about epidemiological studies.
But first I can’t resist pointing out that Bernard apparently decided to lead with a favorite technique of the anti-vaccine movement (not to mention much of the alternative medicine movement), namely the pharma shill gambit, which was perfectly encapsulated in the very first paragraph:
This study was funded by CDC and conducted by several parties with an interest in protecting vaccine use: CDC staff involved in vaccine research and promotion; Abt Associates, a contract research organization whose largest clients include vaccine manufacturers and the CDC’s National Immunization Program; America’s Health Insurance Plans, the trade group for the health insurance industry; and three HMOs which receive substantial funding from vaccine manufacturers to conduct vaccine licensing research.
If you can’t attack the design, execution, and conclusions effectively, then attack the funding source. I wondered why Bernard decided to lead with the pharma shill gambit. Then I read the rest of the critique, and I wondered no more. The very nature of the criticisms Bernard makes tells me that she has no understanding of what a case-control study is. As I look at her complaints, I will, hopefully, be able to show you what a case-control study is, at least its basics. Once you understand that, you’ll understand a frequent technique of the anti-vaccine movement used to attack scientific studies: Criticizing them for something they are not.
As way of background, in case you don’t remember or didn’t read my missive last week (in which case, the hopelessly arrogant surgeon within me suggests that it would be a good idea to go back and read it before proceeding further). Now, let’s look at two of SafeMinds’ most revealing criticism (after the pharma shill gambit, of course).
Revealing complaint #1:
The study sample did not allow an examination of an exposed versus an unexposed group, or even a high versus a low exposed group, but rather the study mostly examined the effect of timing of exposure on autism rates. There were virtually no subjects who were unvaccinated and few who were truly less vaccinated; rather, the low exposed group was mostly just late relative to the higher exposed group, ie, those vaccinating on time.
This criticism reveals a shocking lack of understanding of just what a case-control study is. (Well, maybe not so shocking, given the source.) Ms. Bernard is, in essence, criticizing a case-control study for not being a different kind of study. Specifically, she is criticizing this case-control study because it did not compare unvaccinated versus vaccinated children. That’s not how case-control studies work.
Here’s how case-control studies do work. A case-control study is designed to compare a group of subjects who have a condition to a group who do not (in this case, autism and autism spectrum disorders, or ASDs). The idea is to select a group of subjects with the condition and then pick a group of subjects who do not have the condition in such a way that the two groups are as much alike as possible in as many aspects that could be confounders as possible. From a practical standpoint, what investigators do is to take a study population and identify cases. They then take a random subset of cases if they can’t examine all the cases, which is usually the situation. Next, they look at the rest of the population and randomly select subjects, after which they pick subjects from that randomly selected pool in order to make the case group (those with the condition being studied) as similar to the control group (those without the condition being studied) as alike as possible in all other ways besides the condition under study by matching them for as many parameters relevant to the condition as possible.
The next step is then to look for differences in the group. For a study of this sort, the specific differences being sought is exposure to a substance thought to be causative for the condition being studied. If the cases, for instance, are found to have had a higher exposure to the substance under study, then the conclusion of the case-control study is that exposure to the substance in question is associated with the condition under study and therefore the substance in question might cause or contribute to the condition. If the exposure to the substance under study is lower in the case group than in controls, then the conclusion of the case-control study would be that higher exposure to substance under question is associated with the control group, suggesting that exposure to the substance in question is negatively correlated with the condition in question. Such a result may indicate that the substance in question is actually protective against the condition being studied. Finally, if exposure to the substance in question is found to have been the same between the groups, then the conclusion is that that substance probably has no relationship to the condition under study.
That’s exactly what this study found: that there was no difference between thimerosal exposure between cases and controls. More specifically, it found that there was a somewhat higher exposure to thimerosal among controls, hence the barely statistically significant hazard ratio suggesting a protective effect. Because the investigators did not have a biological mechanism that could explain such an effect, they looked at potential sources of bias and could find no obvious ones. Now, in light of the knowledge I’ve just imparted, take a look at a couple of comments. First, there is Jake Crosby:
“Ms. Bernard is, in essence, criticizing a case-control study for not being a different kind of study.”
That’s absurd, even in a case-control study you should be able to do basic comparisons of exposed and unexposed across the two groups.
Uh, no, Jake. Usually not. That’s not how case-control studies are usually designed or statistically powered because case-control studies are all about comparing cases with controls, not controlling exposed with unexposed.
Then there’s this complaint in the comments on fellow SBM blogger Scott Gavura’s post about this study on Science-Based Pharmacy:
If controls have identical exposure to cases a case control study will tell you nothing.
Do you accept that. It is fundamental to being able to make any comparison between cases and controls.
This comment was from someone going by the ‘nym of childhealthsafety, who kept repeating the same complaint even after several other commenters tried to explain to him what a case-control study is, as commenter Adam does here:
As I have tried very patiently to explain to you before, if controls have identical exposure to cases, it tells you that the exposure is not associated with the outcome.
It may not be the message you want to hear since it doesn’t fit in with your anti-vax prejudices, but that doesn’t mean that it’s in any way invalid.
Sallie Bernard’s complaint is only a more subtly done version of childhealthsafety’s and Jake Crosby’s complaint. All are founded on a mistaken idea of what a case-control study is. Their arguments make me suspect that they do not know the difference between a case-control study and a cohort study. A simple way of looking at the difference is that a case-control study compares subjects with a condition to those without and tries to identify factors that are different between them (in the case of Price et al, thimerosal exposure), while a cohort study looks at subjects without the condition being studied, divides them into groups based on putative risk factors for that condition, and then asks the question whether more subjects with those putative risk factors develop the condition under study. A key feature is that none of the subjects initially have the condition being studied. Both designs have strengths and weaknesses. One strength of a cohort design is that it can be undertaken prospectively, although a prospective cohort study is no longer possible for thimerosal exposure as a risk factor for autism because thimerosal was removed from childhood vaccines by early 2002, which is far longer than the interval between birth and the first symptoms of ASDs, which is usually around 3 years. In any case, basically Bernard, Jake, and childhealthsafety are criticizing a case-control study for not being a cohort study, as much as Jake tries to deny it.
But it’s worse than that. Check out Bernard’s next criticism:
Another validity problem is the effect on exposure variation of stratifying/matching by birth year and HMO. No reason was provided for matching based on year of birth since the long follow up period allowed sufficient time for all cases to be diagnosed. The matching requirements lead to two statistical problems.
I’m no epidemiologist (and if there are any epidemiologists out there interested in writing for SBM, please contact me), but even I know that matching by birth year is such an utterly accepted and conventional method of matching for case-control studies, particularly those studying children, that Bernard’s complaint is, to say the least, perverse. Matching by case year minimizes the variation between cases and controls that might be due to being raised in different years, going to the same schools in different years, or having different exposures related to being born different years. In other words, this is a non-complaint, and Bernard’s reason for it simply adds to the revelation of her complete lack of understanding of what a case-control study is:
Each of the three HMOs would buy in bulk the same vaccines for all its patients and the promotion of a new vaccine would tend to be uniform across an HMO, so that within an HMO, exposure variability is lessened. Additionally, the recommended vaccines, the formulations offered by manufacturers, and the uptake rate of new vaccines varied by year, so that within a given year, exposure variability is further reduced. The effect is that children in a given year in a given HMO would tend to receive the same vaccines.
She writes this as though it would a bad thing for the study to minimize as many differences as possible between the two groups in a case-control study! After all, the study variable is thimerosal, not vaccines. If you want to concentrate on thimerosal, then naturally you’d want to eliminate as many of the other variables as possible, including the number of vaccines each child received. Again, the hypothesis is that it is thimerosal in the vaccines that causes autism, not that vaccines cause autism (although certainly Sallie Bernard also believes that and seems unable to keep from confusing the two hypotheses in her mind). Matching by birth year is one way to help accomplish that. She also constructs a rather bizarre “what if” scenario:
The variables of time and place (HMO) are correlated with the exposure variable. Statistically, the correlation would reduce the effect of the exposure variable, as the two matching variables compete with the exposure variable to explain differences in the autism outcome. For example, say for simplicity that HMO A used vaccines in 1994 which exposed all enrolled infants up to 6 months of age with 75 mcg of mercury; the rate of ASD for 1994 births in HMO A was found to be 1 in 150. In 1995, HMO A used vaccines which exposed all enrolled infants up to 6 months of age to 150 mcg of mercury; the rate of ASD for these children rises to 1 in 100. By stratifying by year for this HMO, those children born in 1994, whether or not they had an ASD, would show identical exposures. Those with an ASD born in 1995 in HMO A would also have the same exposures as those born in 1995 in HMO A without an ASD. The association between the increased exposure and the increase in ASD can only be detected by removing the birth year variable, which otherwise masks the effect of exposure on outcomes.
Once again, Bernard misunderstands the concept of a case-control study. In a case-control study, investigators divide subjects up into cases and controls and then look for differences between them. Besides her misunderstanding of how case-control studies are done, this is one of those criticisms that sounds superficially plausible–if it weren’t for all the other correlations also tested in the various multivariate models used in this study, such as measures of birth weight, household income, maternal education, marital status, maternal and paternal age, birth order, breast feeding duration, child birth conditions including Apgar score, and indicators for birth asphyxia, respiratory distress, and hyperbilirubinemia; measures of maternal tobacco use, alcohol use, fish consumption, exposure to non-vaccine mercury sources, lead, illegal drugs, valproic acid, folic acid, and viral infections during pregnancy were created, and measures of child anemia, encephalitis, lead exposure, and pica.
Moreover, in multiinstitutional studies or studies conducted at more than one site, it is considered mandatory to compare the characteristics of the subjects enrolled at each site in order to make sure that they are comparable and can thus be used in the study. Add to that all the other subject characteristics examined, and Bernard’s complaint becomes just another smokescreen, especially since other results not reported in the Pediatrics paper suggest that autism prevalence was stable during the six years covered. In fact, if you look at the technical report, you’ll find that the authors checked the influence of HMO:
Were overall results driven by results from one particular HMO? To address this question we fit models separately to the data from the two largest HMOs and compared the results to the overall results. The exposure estimates from each of the two large HMOs are similar in direction and magnitude to the overall results. However, they were seldom statistically significant due to the smaller sample sizes obtained when modeling separately by HMO. We conclude that the overall results were not primarily driven by the results in one particular HMO.
They also controlled for study area:
Controlling the geographic area within the HMO coverage could increase the comparability of the cases, as well as make the data collection more concentrated and therefore less expensive. During creation of the sampling frame, children that were known to live more than 60 miles from an assessment clinic were excluded from the sampling frame.
Finally, they did several statistical tests to determine if the results were driven primarily by one subgroup:
In order to assess whether the results were sensitive to the influence of one or a few highly influential observations within a single matching stratum, we tried re-fitting the analysis model for the AD outcome to sequential subsets of data where, in each subset, all data from a single stratum were omitted. For example, if one or a few highly influential observations were in Stratum “2″, then results from a model where the data were omitted from that stratum would be very different from the results when the data from the stratum are included.
It is in general a good idea at least to peruse the entire technical report before making complaints like this. True, the reports are several hundred pages, but there is an incredible amount of detail about the design of the study and how the data were analyzed, far more than could be fit into any paper. It’s rare to have a study for which so much information regarding the nitty-gritty of its design and analysis are made publicly available. Another complaint is that there was a low response rate, which is the same complaint Bernard made about Thompson et al three years ago. The answer is the same now as it was three years ago, and it is also included in excruciating detail in the technical report.
IF YOU CAN’T FIND A DEFICIENCY IN A STUDY, MAKE ONE UP!
Case-control studies are, by their very nature, retrospective studies. That means that they can have confounding factors resulting in false positives or false negatives. However, because it is unethical to do a randomized, double-blind clinical trial to address the question of whether thimerosal exposure due to thimerosal-containing vaccines causes autism, science-based medicine has to make do with the highest quality retrospective data that can be obtained. Price et al is simply the latest piece of data that uses retrospective data to ask whether thimerosal is safe and has come up with the answer that it thimerosal-containing vaccines did not cause an “autism epidemic.” Moreover, one thing that Price et al has done, which Bernard notices, is that is not the only study in which thimerosal exposure was associated with a decreased risk of ASD, which brings her back to the apparent protective effect from thimerosal.
Amazingly (well, not so amazingly), she doesn’t note that the authors acknowledged and discussed this result. She then constructs a scenario designed to “demonstrate” that shifts in participation in key groups in such a study can change the results. No kidding. Here’s the problem. Although Bernard does show that differences in the rate of participation in the controls based on whether they are late vaccinators or not could change the ratio of late vaccinators to have a higher percentage of “vaccinators,” this is yet another smokescreen. For one thing, she envisions identical participation rates between late and on-time vaccinators in the ASD group, while in the non-ASD group she envisions 40% participation of on-time vaccinators and only 15% participation of late vaccinators. This is, to say the least, a highly artificial and unlikely construct, but that’s what it took for her to make the numbers work. To justify these numbers, she cited a paper in which the response rate for subjects with no thimerosal exposure was 48% and those with “full exposure” was 65%. That is not a nearly three-fold difference.
In other words, Bernard had to make up a highly artificial hypothetical situation in which she came up with differences far beyond what is justified in order to make the numbers in her scenario work. Nowhere does she show that there’s any reason to suspect such a huge difference in response rates. Certainly, I could find no indication that would lead me to suspect such huge reporting differences. If that’s the best she could come up with, Price et al is a better study than I thought the first time around.
As desperate as it is, though, what Bernard does with regards to “what if” scenarios of extreme numerical prestidigitation designed to “prove” that, if the phase of the moon is right and huge percentages of parents in different groups declined to participate and did so in just the right way to cook the numbers, a harmful effect due to thimerosal can be turned into a protective effect is nothing compared to what childhealthsafety does in his criticism of the study:
It is known that children with autistic conditions have difficulty excreting mercury [some references below]. The mercury accumulates in their body tissues including the brain, unlike their non autistic counterparts. Mercury is highly neurotoxic — in parts per billion. Only infinitesimally tiny amounts can do significant damage to a developing infant brain.
Despite this being known and documented, the authors of this Pediatrics paper simply measured how much mercury went into all the children but not what did or did not come out. No information was obtained about how much mercury the autistic children accumulated in their brains compared to the amounts excreted by the non autistic comparison group of children. End result — another piece of hyped junk science.
The cases of autistic children were not matched with a comparable group of non autistic “control” children to enable a proper comparison to be made. Yet the study was supposedly a “case-control” study. For the cases to be matched to controls it would be necessary to check the controls retained mercury in the same manner as the autistic cases.
Those who have been involved in the vaccine wars as long as I have will immediately recognize this as the Legend of the “Poor Excretors.” There is, of course, no evidence for the concept that children with autism and ASD have any more difficult excreting mercury than anyone else. It’s not for nothing that this canard has been referred to as the myth of the “poor excretor.” Of course, that hasn’t stopped the pseudoscientists from trying again and again to show that there is somehow a huge difference between autistic children and neurotypical children in how they handle mercury. As far as science has been able to tell, there isn’t. Given that there is no good scientific justification to match controls with cases on the basis of mercury excretion, Price et al didn’t do it. Besides, to match children on such a basis would require that all the cases and controls had been tested for various measures of mercury excretion. Given that such tests are relegated to the realm of DAN! doctors and their pseudoscience, it would be highly unlikely that such data would be available anyway. This is, after all, a retrospective study. I am, however, honored that childhealthsafety would decide to name me in his other attack on the study.
CONCLUSION?
As much as I would like to think that Price et al would be the last word, given the number of large, well-designed studies that preceded it and found no association between thimerosal in vaccines and autism. This hypothesis has been thoroughly studied and found to be wanting. Indeed, beginning three years ago, the weight of the scientific evidence against the mercury-thimerosal-autism hypothesis was such that even Generation Rescue had moved from claiming that autism is a “misdiagnosis for mercury poisoning” to the vaguer (and much more difficult to falsify) hypothesis of “too many [vaccines] too soon” as a cause of autism. Yet, the mercury hypothesis is the zombie that won’t die. After having made my Zombieland-reference to the “double-tap” in my last post as being necessary to make sure that this particular zombie stays dead, unfortunately it’s become clear to me that a double-tap isn’t enough. Neither is a triple-tap or a quadruple-tap. In fact, even my reference to using a .50 caliber machine gun to the point where this hypothesis has been ripped into tiny shreds would probably not be enough. Sadly, given the tenacity with which the anti-vaccine movement clings to the pseudoscience claiming that mercury in vaccines causes autism, I doubt even a thermonuclear blast would be enough to kill this hypothesis off.
However, if we look at the actual arguments used by anti-vaccine zealots and carefully dissect them, it is possible to salvage some science education out of pseudoscience.

   Pseudo-expertise versus science-based medicine
David Gorski 
Link
I am a skeptic.
My support for science-based medicine, as important as it is and as much time, sweat, and treasure I spend supporting it, is not the be-all and end-all of my skepticism, which derives from a scientific world view. That’s why, every so often, I like to step back from medicine a bit and look at the broader picture. It’s a good idea to do this from time to time, because to me, many of the topics that I and my fellow SBM bloggers write about are not just manifestations of anti-science and pseudoscience in medicine, but rather of a broader problem of anti-science and pseudoscience in society at large. I concentrate on medicine because it’s what I do and because manifestations of pseudoscience in medicine have the potential to harm or even kill large numbers of people.
Look no further than the anti-vaccine movement if you don’t believe me. Already, a mere decade after Andrew Wakefield’s lawyer-funded, incompetent, and perhaps even fraudulent “study” about a supposed relationship between the MMR vaccine and autism, uptake of MMR vaccines have plummeted throughout the U.K., with some areas of London reporting only 50% uptake, far too low for effective herd immunity. Thanks to J.B. Handley, Jenny McCarthy, Jim Carrey, and the know-nothing band of celebrities and activists, we are in serious danger of having the same sort of thing happen right here in the U.S. Indeed, Jenny McCarthy herself has even acknowledged that, although in her characteristically self-absorbed and vulgar manner, she refused to take responsibility for her part in this impending public health debacle, dismissing her role by saying, “I do believe sadly it’s going to take some diseases coming back to realize that we need to change and develop vaccines that are safe. If the vaccine companies are not listening to us, it’s their fucking fault that the diseases are coming back. They’re making a product that’s shit. If you give us a safe vaccine, we’ll use it. It shouldn’t be polio versus autism.” Meanwhile autistic children suffer from the quackery to which they are subjected in a futile attempt to “recover” them from “vaccine injury”-induced autism.
But it’s not just the anti-vaccine movement. It’s cancer quackery, promoted by “luminaries” such as Suzanne Somers and Bill Maher, given aid and comfort by doctors gone bad such as Dr. Rashid Buttar and Dr. Nicholas Gonzalez, “bioidentical hormone” woo promoted by the aforementioned Suzanne Somers and Dr. Christiane Northrup. It’s all manner of other faith-based and definitely non-science-based medicine so called “complementary and alternative medicine” (CAM, which is neither complementary nor medicine, although there is no doubt that it’s “alternative”) or “integrative medicine” (which “integrates” pseudoscience with effective medicine to the detriment of patients) finding its way into our academic medical schools, even to the point of being mandatory at at least one medical school and being a strongly touted option at many others. Meanwhile, the misbegotten behemoth of woo, funded by your tax dollars and mine, the National Center for Complementary and Alternative Medicine (NCCAM) promotes remedies based on a prescientific understanding of how the body works and what causes diseases, even going so far as to promote “integrative medicine” residencies. Meanwhile science-based medical students face a serious dilemma: Go with the flow or fight.
Although I’ve only been a blogger for SBM for less than two years and its managing editor for less then a year, I have been in this battle for nearly a decade in various forms, and not just in medicine. Indeed, over the last several years, I’ve subjected myself to some of the most outrageous bits of unreason, conspiracy mongering, and pseudoscience. Be it the anti-vaccine movement, quacks, 9/11 Truthers, Holocaust deniers, creationists, or any of a variety of other bits of pseudoscience, I’ve come to appreciate that what distinguishes believers in such nonsense seems to be, as Prometheus so aptly put it, the arrogance of ignorance. Even so, there seems to be more than that going on, and leave it to, of all things, an article in the L.A. Times by James Rainey entitled Childhood vaccines, autism and the dangers of group think. It’s an article looking at Amy Wallace’s excellent article for WIRED entitled An Epidemic of Fear: How Panicked Parents Skipping Shots Endangers Us All, which documented how the arrogance of ignorance has led the anti-vaccine movement to endanger public health, and the exceedingly (and typically) nasty reaction the anti-vaccine movement with which the anti-vaccine movement responded, particularly J.B. Handley’s misogyny.
There are two key passages in Rainey’s article that tell the tale, a tale that is no surprise to skeptics, in particular skeptical bloggers like myself:
“They will say, ‘Who do you think you are to tell me?’ or ‘Who does the government think it is to tell us what is best for public health?’ ” Wallace told me this week. “They say, ‘You can’t know my child like I know my child.’ ”
Wallace has run smack into an abiding, perhaps growing, phenomenon of the Internet Age: Citizens armed with information are sure they know better. Readers who brush up against expertise believe they have become experts. The common man rebels against the notion that anyone — not professionals, not the government and certainly not the media — speaks with special authority.
Where it stops, nobody knows. But already we see a wave of amateurs convinced they can write a pithier movie review, arrange a catchier song, even assess our planet’s shifting weather conditions, better than the professionals trained to do the job.
I wish he hadn’t mentioned movie reviews and song-writing. Not to denigrate either skill, one of which I can do (albeit not that well) and one of which I can’t (I’ll leave it to the reader to guess which is which), but in contrast to examining the complex science behind medicine, it is much more possible for an amateur to succeed in these endeavors than it is for an amateur to succeed at analyzing the complex problems science addresses at the highest level of medicine and other sciences. Indeed, writers penned books and articles, and songwriters wrote songs before there ever was such a thing as a “professional” songwriter. Come to think of it, people did science before there was such a thing as a professional scientist, too. The difference, at least today, is that science has become so technical and complex that it is very, very difficult to master and all but impossible to teach oneself, particularly medicine.
However, Rainey’s better at identifying a major contributing factor:
The rise of computer literacy, high-speed Internet connections, blogging and social networks has emboldened the common man to tell his own story and, sometimes, to disdain trappings like a university degree, professional training or corporate affiliation. The citizen activists often frame themselves as truth tellers fighting against an establishment that is hopelessly venal. No matter that the corruption, routinely claimed, is seldom supported by more than innuendo.
This is indeed the cult of the amateur, as the title of a book mentioned in the article goes. There has always been a strain in American culture that is deeply anti-intellectual and suspicious of experts, as documented in Richard Hofstadter’s Anti-Intellectualism in American Life back in 1966, with the same lament being echoed by Susan Jacoby last year in her The Age of American Unreason. Although it can cause a lot of problem, a healthy skepticism of experts is not necessarily a bad thing. Experts are not always right, and “the best and the brightest” have at times led us horribly astray. However, in the process, our nation appears to have somehow devalued not only expertise, but science itself. Science is the “other.” It’s not something that “everyday people” do, or at least it’s not perceived that way, which is all the more sad because anyone with a reasonable level of intelligence should be able to understand the very basics of the scientific method. The same is true of critical thinking. Indeed, in many areas of of life, the “average Joe” is admirably skeptical. For example, many people are more than capable of evaluating the sales pitch of a car salesman or, as my wife and I had to do several months ago, the high pressure sales pitch of a roofing salesman. Yet, in other areas these same people are credulous marks for any conspiracy theory that comes around.
In the case of the anti-vaccine movement, what drives this arrogance of ignorance is an old-fashioned American distrust of authority (often good, but not always) combined with a democratic tradition in which every person is assumed to be equal. The problem is that equality before the law and possessing equal rights (which are the American ideal) do not equate to equal abilities or knowledge. Unfortunately, we as a people seem to conflate the two and assume all too often that, if Paul Offit can pontificate about vaccines, so can we, even though we don’t have any special expertise in the relevant sciences about them. Too many of us assume that a few days or a few hours (or, sometimes, even much, much less) spent in front of a computer studying at the University of Google renders our understanding nearly equal to (or even greater than) that of scientists and experts who have spent their entire lives studying a problem. Celebrities are no different, either. Indeed, fueled by ego and surrounded by yes-men and other enablers, celebrities seem even more prone to the arrogance of ignorance, be they Bill Maher, Oprah Winfrey, Jenny McCarthy, Jim Carrey, Billy Corgan, or Suzanne Somers. Worse, they have a much larger soapbox from which to spread their nonsense. But they’re not alone. Whenever I want to demonstrate what drives this attitude, I like to quote anti-vaccine activist, bully, and all-around unlikable guy J.B. Handley:
I’m not intellectually intimidated by any of these jokers. Their degrees mean zippo to me, because I knew plenty of knuckleheads in college who went on to be doctors, and they’re still knuckleheads (I also knew plenty of great, smart guys who went on to be doctors and they’re still great, smart guys).
I chose a different path and went into the business world. In the business world, having a degree from a great college or business school gets you your first job, and not much else. There are plenty of Harvard Business School grads who have bankrupted companies and gone to jail, and plenty of high school drop-outs who are multi-millionaires. Brains and street-smarts win, not degrees, arrogance, or entitlement.
Except that brains and street smarts count for very little in science if they exist without an understanding of the scientific method and knowledge of the context behind a scientific study or even a scientific consensus on an issue.
From my perspective, the progress made on developing Internet may well be the single greatest development of the last 30 years. When the Internet was first developed, it was used primarily by educational, government, and defense institutions. It wasn’t until the mid 1990s when huge numbers of people started to have access to the Internet, and today in developed countries most people take Internet access for granted. Personally, I don’t know how I’d survive without it. It’s made, for example, looking up articles for my research and writing journal articles and grants a snap, comparatively speaking. However, there’s a down side, and that’s too much information, so much information that it makes it very easy for someone without the requisite background necessary to be able to separate the wheat from the chaff to develop a sense of pseudo-expertise. In other words, they may pick up a lot of facts and be able to cite a lot of studies, but they do not know the scientific context behind these facts and studies. They don’t know why scientists will value one study over another or consider the studies they like to cite to be nothing more than a load of fetid dingo’s kidneys, not worth the electrons that carry the PDF file containing the study to the computer the crank used to download it. Worse, they don’t know how to recognize good studies compared to bad studies or understand that critically examining the evidence against your beliefs is even more important than examining the evidence for them. The result all too often turns into an orgy of cherry picking and confirmation bias.
The result, when combined with someone like J.B. Handley, who thinks that expertise can be so easily dismissed, is the anti-vaccine movement, creationists, Holocaust deniers, 9/11 Truthers, and quacks.
The other driving force behind the proliferation of pseudo-expertise is a very human trait that we all share, namely the tendency to confuse correlation with causation. Once again, this is one of the first lessons in science, not to confuse correlation with causation, but those of us in science forget just how against human nature this is. We are creatures that value personal experience over statistics and science. One good anecdote trumps reams of evidence, hundreds of painstaking scientific studies, even a convergence of evidence from multiple different disciplines. This produces, for example, the alternative medicine maven who swears by homeopathy because the symptoms of their self-limited condition got better after they tried it, anthropogenic global warming denialists who justify their rejection of climate science by their observation that this summer was unusually mild in their area, and deniers of evolution who look at nature and cannot fathom how humans might have evolved from other animals over millions of years because they an only see the pitiful handful of decades each of us is given to walk this earth. It’s what drives Suzanne Somers to believe that her decision to eschew chemotherapy in favor of mistletoe extract after her curative surgery and radiotherapy is what saved her life. It’s what drives virtually the entire groundswell of support for CAM modalities, the vast majority of which do whatever it is they do through placebo effects.
Having said that, let us not forget that, at the level of a single person, correlation sure can appear to be causation. As was pointed out a month ago by Orac, one example is heart attacks and the flu vaccine. More than 3,000 people have heart attacks each and every day, which means that by random chance alone there will be probably several people a day who have a heart attack within 24 hours of being vaccinated for the flu. “On the ground,” to those people it may appear all the world as though the vaccine caused the heart attack, when in reality it really was just coincidence. It’s not enough simply to observe an adverse event happening after something, say, vaccination. You have to show that there is an incidence of that adverse event significantly greater than what could be predicted by chance alone. The same applies to the claim that vaccines cause autism. If you have a child who regresses within a day or so of vaccination, it will appear all the world to you that the vaccine caused the regression. In that case, it is then very difficult even for highly educated parents to accept the results of science, namely that epidemiological studies do not find an elevated incidence of autism after vaccination.
Combine the all-too-human tendency to confuse correlation with causation with the anti-intellectual attitude of a J. B. Handley, a Jenny McCarthy, a Suzanne Somers, a Bill Maher, or any of the panoply of celebrities, celebrity wannabes, or regular people who made it big through the promotion of snake oil with the arrogance of ignorance that pseudoexpertise derived from studying at Google U. produces, and you have fanatical adherence to a crank movement. It all boils down to a basic human need for a perception of order in the universe. We need to identify causes when bad things happen; we need explanations. To the non-scientist (and even to all too many physicians and scientists), “you were unlucky” or “it was just an unfortunate coincidence” are not answers to the question “Why?” that satisfy. They may be the most likely to be true, but they do not satisfy. Blaming something does, be it blaming vaccines for autism or constructing elaborate conspiracy theories to explain how 19 men with box cutters could hijack commercial airliners, destroy the two of the tallest buildings in the U.S., severely damage the Pentagon, and cause the deaths of 3,000 Americans, or why evolutionary biologists do not accept that “God did it” as an adequate explanation for the diversity of life in direct opposition to what preachers and parents have taught.
Becoming a real expert in anything is very hard. As several surgical conferences and Grand Rounds have recently reminded me, it’s been estimated that in general it takes 10,000 hours of practice and study to become an expert in surgery. There are no shortcuts around that. True, it may take somewhat less for surgeons with a natural knack for operating and patient care and longer for those of more modest and average capabilities, the latter of whom often have to work harder and practice longer to achieve the same level of expertise (if they ever achieve it at all), but that’s the average number of hours needed to become a master. The Internet may seem like a shortcut that levels the playing field between experts and the great unwashed masses, but in reality it only gives an illusion of expertise or, as I’ve called it, pseudoexpertise. Similarly, in the past, the lay person just plain did not have direct access to medical studies. Obtaining such studies would require a trip to a medical school library, which may or may not be far away, prolonged searching through Index Medicus, piling journal upon journal on a cart, and then feeding coin after coin into the machine to copy the articles desired. Now, virtually any abstract can be accessed through PubMed, and articles reporting federally funded research are deposited in PubMed Central within a year of publication, where anyone can access them. While this open access to knowledge is appropriate, given that our tax dollars funded the research, it inadvertently fueled the rise of the pseudoexpert.
It’s also been pointed out before that medicine seems particular prone to the depradations of the pseudoexpert. People who would never think trusting “alternative flight” or “alternative engineering” seem to think that, when it comes to medicine, “alternative” medicine should be given a pass on being as rigorously reality- and science-based. Worse, I’ve seen this tendency even in people who really, really should know better. For example, over the last several years, at times I’ve seen a distressing tendency for even those proclaiming themselves to be “skeptics,” even going so far as to join skeptical organizations like the Center For Inquiry or to support skeptical organizations like the James Randi Educational Foundation, to be prone to some serious woo when it comes to medicine. I’ve spoken with self-proclaimed “skeptics” who actually buy into the idea that vaccines somehow cause autism, a pseudoscientific concept that has so much evidence against it that personally I would almost go so far to say that anyone who takes such anti-vaccine propaganda seriously should forfeit the title of “skeptic.” (Actually, forget the “almost.”)
Perhaps the most striking example of this disconnect occurred when the Atheist Alliance International (AAI) bestowed the Richard Dawkins Award upon a rabidly anti-vaccine comedian named Bill Maher. True, Maher holds the “right” position with respect to religion (he’s against it) and science such as anthropogenic global warming (he accepts the science supporting it and ridicules AGW denialists), and evolution (he accepts the science and ridicules creationists). It also appears in retrospect to be equally true that, at the time the AAI chose him for the Richard Dawkins Award, it apparently didn’t matter one whit that Maher is simultaneously as rabidly anti-science as Jenny McCarthy, Suzanne Somers, or Jim Carrey when it comes to medicine, championing anti-vaccine views and distrust of “Western” medicine (which is, as readers of this blog know, usually a derogatory code word for science-based medicine), cancer quackery, and germ theory denialism, even though one of the criteria for the award was that its recipient must “through writings, media, the arts, film, and/or the stage advocates increased scientific knowledge.” It only mattered later, when complaints about the selection started to bubble up too insistently to be ignored.
Truly, even though there are notable and heartening exceptions, such as Phil Plait, James Randi, and our very own Steve Novella and Harriet Hall, all too often science-based medicine is treated like the Rodney Dangerfield of the skeptical movement. Certainly, compared to evolution, we “don’t get no respect.” Or so it all too often seems to me. At the very least, until relatively recently, we didn’t get the same level of attention as creationists, dowsers, and psychics. Even now, with things probably better on that score, I’m still not sure that we do.
It’s not all bad, though. The very same forces that produced the anti-vaccine movement and fuel the panoply of cranks provide the weapons to combat them. It is that easy access to blogs and the web that cranks take advantage of to spread their message that provides scientists and skeptics the weapons to combat cranks. The Internet, while empowering all manner of pseudoscientists, snake oil salesmen, and enemies of public health such as anti-vaccine activists, also allows skeptics and supporters of science-based medicine to band together to counter the madness. That’s what we try to do right here each and every day, to provide a voice raised against the anti-science and pseudoscience engulfing medicine. Unfortunately, it’s a lopsided battle, and not lopsided in our favor. As long as there is Oprah Winfrey, Jenny McCarthy, Jim Carrey, Bill Maher, Suzanne Somers, and other celebrities brimming with the arrogance of ignorance, it will be a lopsided battle. But it’s not by any means a hopeless battle, and it’s one we can’t afford to lose, even though none of us will live to see a victory against pseudo-expertise.

The Misinformation Dilemma
Is there a workable solution to the social media dilemma?
Steven Novella on January 19, 2022
Link

Institutions are still struggling with the challenge of dealing with deadly healthcare information. This challenge was not caused by social media, but certainly was exacerbated by it. And it also did not suddenly become relevant during the COVID pandemic, but was brought into sharper relief. What we are experiencing now is actually part of a much longer trend, and is essentially a conflict between freedom on the one hand and quality control for the purpose of public good on the other. A solution at either extreme seems unlikely, and perhaps there is no ideal solution. But there still may be some practical things we can do to live with a compromise in the middle.
Democratization of information
The easy and free spread of information did not begin with social media or the internet. In fact the same angst many are feeling toward the uncontrolled spread of information was felt when the printing press was introduced. The printing press, however, is also credited with the European Renaissance and the Scientific Revolution (at least as one major factor). Thomas Paine’s 47-page tract was also a powerful influence on the public support for declaring independence from England.
Over the last few centuries technological advance has led to increasing ease and speed at spreading information and ideas. With each new technology, telegraph, radio, television, film, the internet, and social media, the same arguments over the benefits and perils of democratizing information resurface. It is arguable that the net effect of democratizing information has been positive (if you consider science and democracy good things). The primary perceived downside was in the ability for commoners to challenge authority, something which we would view from our modern perspective as a good thing. Part of science was destroying the old paradigm of basing claims on authority and substituting it with basing claims on logic and evidence.
Ironically the democratization of information allowed for the scientific revolution, but science itself has now altered the calculus. This is because naked authority has been replaced with genuine and valid mechanisms of quality control. Essentially, from science to journalism and the media in general, traditional authority was bypassed with technology that allowed for widespread dissemination of information. A new cultural balance was achieved, however, with the rise of editorial filters and institutional quality control to replace authority based on history and culture. In a free and open society anyone could make their case, but in order to really spread information you typically had to get past some editorial filter. The real cranks were left shouting on the street corners, but they we generally not getting published or giving lectures at universities.
TV and radio changed the balance further, not by removing the filter (not anyone could get on prime time television) but by blending information and entertainment, until there was no meaningful distinction between the two. In fact what is ostensibly presented as news could be weaponized as entertainment to reach a wide audience and carrying a payload of ideological propaganda. Even without social media, this trend has created social change and an unsettling dilemma. Social media now adds a new layer. First, it obliterates any remnants of an editorial filter. Literally anyone can communicate with a wide audience. The cranks on the corner are given a megaphone. Further, computer algorithms designed to maximize engagement also maximize radicalization, favoring misinformation if anything over quality control.
This seems qualitatively different than the printing press, not just an extension of it.
Freedom vs quality control
At the optimistic “total freedom” end of the spectrum are those who argue that we should have completely unfiltered spread of information, and the marketplace of ideas will sort it out. The cream will rise over time to the top, and we just have to tolerate the noise at the bottom. Any attempt at quality control is really just authoritarian censorship, and should be vigorously opposed.
But recent scientific evidence regarding how information spreads online, and basic observation of the world we are currently living in, makes this view seem unrealistic to the point of being naïve. There are countless surveys showing the degree to which the public believes demonstrable nonsense, based on the misinformation. It’s also clear that social media has given psychopaths and con artists the keys to the kingdom. It now pays, big, with little upfront investment, to spend a lot of time and energy crafting and spreading misinformation online. And not just isolated bits of misinformation, but a web of distortions and lies that are woven into a psychologically compelling narrative. The “cream” that is demonstrably rising to the top is that information which tends to sell products and radicalize its consumers.
The problem with “all quality control is censorship” position is that it is outdated. It is based in a world in which authorities were not legitimate, but were based solely on historical power. But now we have genuine mechanisms of quality control, such as peer review and rigorous science. Those too are eliminated when we remove all filters.
This is the dilemma we currently face – how do we collectively maintain effective quality of information while simultaneously allowing the free spread of information? There is no single answer, but there are a number of options. Always the best option, because there is no real downside, is education. Public education must teach not only scientific literacy but critical thinking skills and media savvy. We each need to have our own skeptical filters. But this type of education is a generational project, and will always be incomplete.
Many experts recommend adjusting social media algorithms so that they do not favor misinformation, and in fact disfavor demonstrably wrong claims. This way people can say whatever they want, but their reach will be limited. This will partly return us to a time when editorial filters limited the spread of nonsense. Anyone could still get on public access TV if they really wanted, but no one was guaranteed the right to air their views to millions.
Still there is the debate about banning harmful misinformation. This debate has been brought to the fore by the pandemic, as there is a body count and considerable social disruption clearly attached to the spread of health misinformation. The question is whether it is reasonable to outright ban information at the extreme end of the spectrum that is both harmful and wrong. Does freedom and democracy really require that massive platforms are given to utter (and harmful) nonsense?
The trick is – who gets to decide what is nonsense. Those arguing against this measure correctly point out that there is a risk here, that authoritarians will use the mechanisms and justifications of banning harming misinformation to censor politically inconvenient information. Also, the very fact of banning information will be used to sow further conspiracy theories about government and other institutions. Is there a compromise or balance here where the worst information can be completely removed from specific platforms?
I think so. If you are convicted of fraud, your right to engage in certain commercial activities can be limited. You can be banned from, for example, trading stocks or selling investments. Conviction is a transparent process, with rules and mechanisms of appeal. We need a similar process that is transparent, based on agreed-upon rules, and with mechanisms of appeal. If someone is “convicted” of spreading harmful misinformation (not political ideas or unpopular ideas – but demonstrably wrong facts) then removing those false and harmful claims may be justified. This already happens to some degree, when it comes to libel, for example. We recognize the harm that libel can do and provide a legal remedy to remove the libelous information from the public view and penalize those who spread it. A similar process and standard can apply to “libeling the truth”.
        ◦ Again, this is all tricky territory. But this is a conversation we need to have. Recent changes in mass media have destabilized our society, and we need to find some way to achieve a new equilibrium if we are going to be able to deal with modern challenges, like a deadly pandemic.

Yet more evidence that we physicians need to clean up our act
A recent study found that physicians and scientists who are perceived as “experts” are prevalent within the antivax community and more influential because of their status as physicians and scientists. Why do physicians continue to tolerate antivax quacks within our ranks?
David Gorski on February 12, 2024
Link

A couple of years ago, I wrote about how so many of my colleagues seemed shocked at the emergence of so many antivax physicians, even asking: Why is anyone surprised that there are so many antivax physicians? At the risk of coming across as saying, “I told you so,” I basically told you so then, noting that those of us who have been writing about the antivaccine movement knew that none of this was new, that, contrary to what so many of my colleagues (and, truth be told, most pundits) appeared to think, prior to the pandemic far too many physicians held antivax views, a number that has only increased since 2020. Basically, the prevalence of antivax physicians only seemed new to my colleagues because they hadn’t been paying attention before the pandemic and had thus been able to falsely reassure themselves that the antivax quack problem in our profession was just Andrew Wakefield and maybe a handful of cranks and quacks. Of course, my response to that dismissal was always to ask why state medical boards had long been so toothless—and they have long been toothless—with respect to policing our profession and disciplining antivax quacks—indeed disciplining all quacks, which state medical boards have long failed to do (e.g., see the case of Dr. Stanislaw Burzynski, who has been preying on desperate cancer patients since the late 1970s, mostly unmolested). COVID-19 just laid bare how toothless state medical boards have long been with respect to protecting the public from quacks, antivax or otherwise.
“You really think it’s all new.“You really think about it too,”The old man scoffed as he spoke to me,“I’ll tell you a thing or two.“The Clash, Something About England (1981)
Last week, a study further reinforcing just how badly physicians as a profession have failed our patients and society was published in PNAS Nexus, Perceived experts are prevalent and influential within an antivaccine community on Twitter. Ironically, the study’s lead author, Mallory J. Harris, is affiliated with the Department of Biology at Stanford. Why “ironic”? Simple. As we have documented here time and time again, since the pandemic, Stanford University has become one of the foremost institutions harboring prominent academics who have minimized the seriousness and threat of COVID-19, advocated for “let ‘er rip” approaches to the pandemic in a futile effort to achieve “natural herd immunity” (Great Barrington Declaration, anyone?), and even promoted outright antivaccine misinformation. Whether it’s formerly admired academic (by me, anyway) John Ioannidis grossly underestimating the infection fatality rate (IFR) of SARS-CoV-2, accusing doctors of “just going crazy and intubating patients who did not have to be intubated” (and refusing to admit his mistake), and later being reduced to treating a satirical publication index as though it were real in order to attack his critics or Dr. Jay Bhattacharya championing “natural herd immunity” as the solution to the pandemic, Stanford has not exactly covered itself with glory over the last four years. Dr. Bhattacharya, in particular, has gone full quack. He now has a podcast, The Illusion of Consensus, in which he gives credulous interviews to antivaxxers and COVID-19 quacks. He has even featured “debates” on his website between antivaxxers over whether COVID-19 vaccines have killed 17 million, as well as interviews with Kevin McKernan (a key promoter of the “DNA contamination” of COVID-19 vaccines narrative) and Jessica Rose, known for dumpster-diving into the VAERS database to dredge up all sorts of false “injuries” and deaths due to COVID-19 vaccines.
So it’s great to see something fruitful coming out of Stanford, specifically an attempt to quantify how prevalent antivax physicians are on one social media platform, specifically, X, the platform formerly known as Twitter and that the Stanford Center for Innovation in Global Health is still fighting the good fight.
You can also read the Q&A I did on the project & what it means here: https://t.co/x0BXH5cqny
— Mallory Harris (@malar0ne.bsky.social) (@malar0ne) February 7, 2024
Let’s see what Harris and coinvestigators found.
Physician experts on both sides, but with a big difference
From my perspective, what I welcomed about this study is that it tried to quantify a phenomenon that I had long been aware of going back to the dawn of social media, specifically the prevalence of antivax “experts,” legitimate physicians and scientists who, for whatever reason, had gone down the antivax conspiracy rabbit hole and as a result started spending a lot of time online promoting antivax misinformation. (Before social media, it was blogs and even Usenet, as the case of “quack tycoon” Dr. Joe Mercola demonstrates.) Obviously, among these Andrew Wakefield is the granddaddy of them all, at least in the modern antivax movement, although rabidly antivax physicians like “medical heretic” Dr. Robert S. Mendelsohn predated Wakefield by decades.
In the introduction, Harris et al note:
Information consumers often use markers of credibility to assess different sources (25, 26). Specifically, prestige bias describes a heuristic where one preferentially learns from individuals who present signals associated with higher status (e.g. educational and professional credentials) (27–29). Importantly, prestige-biased learning relies on signifiers of expertise that may or may not be accurate or correspond with actual competence in a given domain (25, 30). Therefore, we will refer to perceived experts to denote individuals whose profiles contain signals that have been shown experimentally to increase the likelihood that an individual is viewed as an expert on COVID-19 vaccines (31), although credentials may be misrepresented or misunderstood (user profiles may be deceptive or ambiguous, audiences may not understand the domain specificity of expertise, and platform design may impair assessments of expertise if partial profile information is displayed alongside posts). We focus on the understudied role of perceived experts as potential antivaccine influencers who accrue influence through prestige bias (4, 13). Medical professionals, biomedical scientists, and organizations are trusted sources of medical information who may be especially effective at persuading people to get vaccinated and correcting misconceptions about disease and vaccines (29, 32–35), suggesting that prestige bias may apply to vaccination decisions, including for COVID-19 vaccines (36, 37).
I note that this study examines both antivax “experts” and legitimate experts who are science-based and thus provaccine. The authors note that no systematic attempt had yet been made before their study to quantify the effect of these antivax “experts” on influencing the public, pointing out:
In addition to signaling expertise in their profiles, perceived experts may behave like biomedical experts by making scientific arguments and sharing scientific links but also propagate misinformation by sharing unreliable sources. Antivaccine films frequently utilize medical imagery and emphasize the scientific authority of perceived experts who appear in the films (42, 43, 48). Although vaccine opponents reject scientific consensus, many still value the brand of science and engage with peer reviewed literature (49). Scientific articles are routinely shared by Twitter users who oppose vaccines and other public health measures (e.g. masks), but sources may be presented in a selective or misleading manner (40, 49–53). At the same time, misinformation claims from sources that often fail fact checks (i.e. low-quality sources) are pervasive within antivaccine communities, where they may exacerbate vaccine hesitancy (6, 18, 54, 55). To understand the types of evidence perceived experts use to support their arguments, we ask (RQ2): How often do perceived experts in the antivaccine community share misinformation and academic sources relative to other users?
As I like to say, antivaxxers, especially antivax scientists and physicians, use the peer-reviewed biomedical literature the same way that an inebriated person uses a lamppost at night, for support rather than for illumination. As we here at SBM have discussed time and time again, they frequently misinterpret and/or cherry pick studies to support their position, often highlighting low quality studies from antivax “investigators” published in dodgy, bottom-feeding or even predatory journals.
Before we get to the actual meat of the study, I can’t help but highlight one more quote from the study about why it matters, why “experts” are so influential:
In general, perceived expertise may increase user influence within the provaccine and antivaccine communities. Although vaccine opponents express distrust in scientific institutions and the medical community writ large, they simultaneously embrace perceived experts who oppose the scientific consensus as heroes and trusted sources (10, 41, 43, 48). Medical misinformation claims attributed to perceived experts were some of the most popular and durable topics within misinformation communities on Twitter during the COVID-19 pandemic (12, 63). In fact, compared to individuals who agree with the scientific consensus, individuals who hold counter-consensus positions may actually be more likely to engage with perceived experts that align with their stances (51, 64, 65). This expectation is based on experimental work on source-message incongruence, which suggests that messages are more persuasive when they come from a surprising source (66–68). This phenomenon may extend to the case where perceived experts depart from the expected position of supporting vaccination. In an experiment where participants were presented with claims from different sources about a fictional vaccine, messages from doctors opposing vaccination were especially influential and were transmitted more effectively than provaccine messages from doctors (69).
Medical misinformation claims from “experts” were indeed some of the most popular and, unfortunately, durable topics within misinformation communities during the pandemic. Indeed, if that weren’t the case, we here at SBM probably would have lost half (or more) of our blogging topics, because the list of physician “experts” who undermined anything resembling the scientific consensus has been large and loud, including, for example, Drs. Peter McCullough, Robert “inventor of mRNA vaccines” Malone, William “Turbo Cancer” Makis, and Ryan Cole, the entire crew of physicians at the Frontline COVID-19 Critical Care (FLCCC) Alliance, America’s Frontline Doctors, the Association of American Physicians and Surgeons (AAPS, the granddaddy of all crank and antivax pseudomedical organizations and John Birch Society of medicine, founded in 1943), and others.
In brief, Harris et al had four main research questions:
     1.  How many perceived experts are there in the antivaccine and provaccine communities?
     2.  How often do perceived experts in the antivaccine community share misinformation and academic sources relative to other users?
     3.  Do perceived experts occupy key network positions (i.e. as central and bridging users)?
     4.  Are perceived experts more influential than other individual users?
So, how did Harris et al attempt to quantify the influence of antivax physicians and scientists? In brief, they looked at Twitter during specific time period (April 2021) and collected over 4.2 million unique posts to Twitter containing keywords about COVID-19 vaccines. They then constructed a coengagement network where users were linked if they their posts were retweeted at least ten times by at least two of the same users, meaning that they shared an audience. The resulting network looked like this:
Figure 1. The coengagement network of users tweeting about COVID-19 vaccines is divided into two large communities. Users are represented as circles and scaled by degree centrality. Edges connect users that were retweeted at least 10 times by at least two of the same users. We highlight the two largest communities detected using the Infomap algorithm: antivaccine (pink, lower left) and provaccine (green, top right). Shades indicate account type: excluded from analyses (light); individual perceived nonexpert (medium); and perceived expert (dark). Users outside of the two largest communities are gray. Insets provide more detailed views of: A) the main component of the antivaccine community, B) bridges between the provaccine and antivaccine communities, and C) the main component of the provaccine community. Each edge is colored based on the color of one of the two users it connects, randomly selected. A higher-resolution image without annotations is available as Fig. S1.
The authors then used the Infomap community detection algorithm, which detected a divide in which one group generally posted Tweets with a positive stance supportive of COVID-19 vaccines and the other posted Tweets expressing a negative stance towards the vaccines. The authors, appropriately enough, labeled the first community as provaccine and the second as antivaccine, although the authors did note that the stance “toward COVID-19 vaccines in popular tweets by users with the greatest degree centrality was relatively consistent, meaning that very few users posted a combination of tweets that were positive and negative in stance, although most users posted some neutral tweets as well” and that stances were “generally shared within communities; popular nonneutral tweets by central users in the antivaccine and provaccine communities almost exclusively expressed negative and positive stances, respectively, with a few exceptions.”
Another interesting finding:
The provaccine community was larger than the antivaccine community (3,443 and 2,704 users, respectively). Perceived experts were present in both communities, but constituted a larger share of individual users in the provaccine community (17.2% or 386 accounts) compared to the antivaccine community (9.8% or 185 accounts) (Fig. 3).
I say “interesting finding” because it often seems as though the antivaccine social media community is much larger than the provaccine community, although the authors’ finding that there were more perceived experts in the provaccine community as a percentage of the total community than in the antivax community does jibe with my “gestalt” over social media. That being said, these antivax doctors tend to be very influential.
Another finding that jibes with my experience is that perceived experts in the antivax community frequently share articles from poor quality sources compared to what perceived experts in the provaccine community share. Actually, unsurprisingly, both nonexperts and perceived experts in the antivax community share poor quality sources:
Looking at this figure, I was rather surprised at the low percentage of sources that perceived experts shared that were academic links—in both communities. On the other hand, if you’re trying to influence a lay public, you probably don’t want to be sharing a lot of articles that your target audience might not understand. Be that as it may, it turns out that perceived experts tend to be more influential on social media as measured by their positions as bridging and central users. Moreover, a key observation is that perceived experts in the antivax community appear to share poor quality sources (e.g., domain names that repeatedly fail fact checks) at a frequency indistinguishable from the perceived nonexperts in the same group. I will note that the authors didn’t look at how the academic links shared by the perceived experts in each community were used, but will simply reiterate how in general I have often observed antivax physicians either cherry picking studies and ignoring contradicting evidence, misinterpreting studies to support conclusions the studies don’t support, and/or amplifying bad science done by disreputable “scientists” or physicians published in questionable journals.
As I like to say, the antivax movement in April 2021 could be characterized as the same as it ever was, only more so than before the pandemic. Also remember that April 2021 was only four months after the first COVID-19 vaccines started to be distributed after the FDA granted them emergency use authorization (EUA) in December 2020.
Perceived experts were found to be overrepresented among users with the greatest betweenness, degree, and PageRank centrality in both communities, with the terms meaning (taken from the Supplement):
    • Degree centrality. Number of times a given node appears along the shortest path between two other nodes in the network, which selects users who share audience with users that otherwise do not have much overlap in their audiences.
    • Betweenness centrality. Number of edges connecting a given node to other nodes, which selects users who share an audience with many other users; serves as a proxy for audience size.
    • Page rank centrality. Recursively assigns nodes a value based on whether they are connected to other nodes with high PageRank centrality, which selects users who share audiences with many other users whose audiences overlap with those of many others in the coengagement network.
    • Community bridging. Takes the minimum of the number of edges that a node has connecting it to nodes in any focal communities, which selects users who share an audience with relatively many users in both the anti- and pro-vaccine communities.
It turns out that perceived experts were overrepresented among the users with the highest degree of betweenness centrality, community bridging, as well as degree and PageRank centrality. Perceived experts were overrepresented by a factor of around two in the group of 500 and 50 most central users in the antivax community.
Moreover, when it came to engagement:
Perceived experts in the antivaccine community were 1.43 (95% CI 1.02–1.99) times more likely to receive retweets on their median post than would be expected if they were not perceived experts (Fig. 5 and Table S6). There was no significant effect of perceived expertise on median likes, but there was a significant effect of perceived expertise on h-index for likes and retweets (Fig. 5 and Table S6). Perceived expertise did not significantly affect centrality (betweenness, degree, and eigenvector) within the antivaccine community on average (Fig. 5 and Table S6), although we found in the previous section that perceived experts were overrepresented in the tail of the distribution as highly central users compared to the full (unmatched) set of perceived nonexperts (see Section S9 and Fig. S19).
The authors also compared the size of the influence boost one gets from being a perceived expert in the anti- and provaccine community and found no significant difference, leading them to conclude that perceived experts are roughly similarly important and influential in both communities. What this means is that our profession is simultaneously the most potent tool to counter misinformation as well as to spread it, all because of our status in society, which in this case translates generally to influence on social media.
To me, perhaps the scariest finding discussed is how 28.9% of perceived experts in the two largest communities of the coengagement network examined by Harris et al were part of the antivax community, which is a proportion similar to those reported by other studies of COVID-19 vaccine attitudes expressed by medical professionals on Twitter cited in the discussion (e.g., this study). As the authors note, this suggests that perceived experts who are antivax are overrepresented on Twitter compared to their actual proportion of actual experts; i.e., physicians and scientists.
Moreover unlike Harris, I was not surprised that antivax “experts” were sharing scientific articles online. For instance, in an interview, Harris answers a question about which of the study’s findings they found most surprising:
We saw perceived experts in both the pro-vaccine and the anti-vaccine community sharing many more scientific articles than non-experts. So these perceived experts in the anti-vaccine community did not just have credentials in their names — they were behaving in many of the same ways as their counterparts in the pro-vaccine community, such as discussing scientific papers. 
That’s exactly what I’ve been seeing antivaccine “experts” do online for well over two decades, starting with Usenet, moving to websites and blogs, and then becoming active on social media.
What to do?
So we have a study that suggests that perceived experts, particularly physicians and scientists, are much more influential about COVID-19 vaccines than the average Twitter user, or, at least this was the case in April 2021. Unfortunately, that was before Elon Musk bought Twitter. I say “unfortunately,” because the functional changes made to Twitter, leaving aside the asinine renaming the platform “X,” make a research project like this well nigh impossible to undertake now without great expense. Moreover, given the deterioration of the platform, it wouldn’t surprise me if the situation is much worse now than it was nearly three years ago. Thus, there are a lot of limitations to this study, including that it only looked at individuals discussing COVID-19 vaccines during a given month early during the vaccine rollout, was mainly US-centric and based on English-language Tweets, and used Tweet engagements as an indicator of Tweet popularity (which may lead to an underestimate the true reach of content). Also, Twitter users are most definitely not broadly representative of the general population. Even so, these findings do suggest some issues with commonly believed strategies to combat medical misinformation:
Educational interventions that encourage trust in science could backfire if individuals defer to antivaccine perceived experts who share low-quality sources (76). Instead, education efforts should focus on teaching the public about the scientific process and how to evaluate source credibility to counter the potentially fallacious heuristic of deferring to individual perceived experts (26, 76–78). Although the sample of perceived experts in this study is not representative of the broader community of experts, surveys have found that a nonnegligible minority of medical students and health professionals are vaccine hesitant and believe false claims about vaccine safety (71, 79, 80). Given that perceived experts are particularly influential in vaccine conversations (Figs. 3–5) and that healthcare providers with more knowledge about vaccines are more willing to recommend vaccination, efforts to educate healthcare professionals and bioscientists on vaccination and to overcome misinformation within this community may help to improve vaccine uptake (81).
(I wrote about one of the studies regarding the prevalence of vaccine hesitancy and antivax views among physicians way back in April 2022, by the way.)
Also, Harris said elsewhere:
We have to go a level deeper and question who these people we’re listening to are. What exactly are they an expert in? Have they published on this subject? Might they have financial or political conflicts of interest? We can also try to understand an individual’s reputation in their scientific community. 
All of which are a good start, but not sufficient. If there’s one thing that the pandemic has taught us, it’s that physicians and scientists with stellar reputations can break bad. Again, examples that I routinely cite include John Ioannidis, who, while not being antivaccine, has promoted very bad takes on the pandemic and how to control it through “natural herd immunity“; Dr. Vinay Prasad, who before the pandemic was a respected oncologist but now promotes borderline antivax, if not outright antivax, takes on vaccinating children and boosting adults against COVID-19 to the point of finding some of Robert F. Kennedy, Jr.’s antivax takes compelling; or Dr. Jay Bhattacharya, who has probably by now gone way too far down the antivax quack conspiracy rabbit hole to be salvageable.
I’ve long pointed out that an MD, PhD, or other title doesn’t make one any less human and, as a result, MDs, PhDs, and other experts are just as susceptible to believing pseudoscience and conspiracy theories as nonexpert lay people. As much as we physicians would like to view ourselves as immune—or at least more resistant—to the effects of ideology, emotion, and belief that do not flow from science and reason, we are not. Indeed, it is hubris on our part to think that we are less susceptible to unreason than anyone else, but that doesn’t stop us from thinking that. Worse still, because we as a group tend to think of ourselves as more reasonable and less susceptible to conspiracy theories, misinformation, and disinformation than the average person, when we do fall for them we tend to fall hard, as in very hard. (I’m talking to you, Dr. Peter McCullough and Dr. Pierre Kory, to cite two prominent examples.) Worse still, because we as a group tend to be highly intelligent, we tend to be very, very good indeed at motivated reasoning. Specifically, we are very skilled at asymmetrically seeking out evidence and arguments that support our pre-existing beliefs and finding flaw in evidence and arguments that challenge those beliefs, all while reassuring ourselves that our training as physicians means that our beliefs about medicine are all strictly based on science. That and the public perception of us as experts are two big reasons why we’re so influential on social media when we discuss COVID-19 vaccines. Also, we tend as a group to be overconfident and very good at weaponizing the techniques of scientific analysis in order to “reanalyze” data or “visualize” it in a different and misleading way.
This is a big reason why I emphasize that you need to learn how to tell a likely reliable expert from a quack and try to educate our readers about the warning signs. I grant you that it’s not always easy, as there will always be gray areas. However, most of the time we are not dealing with gray areas, and much of the time distinguishing a legitimate medical expert from an antivax quack can be surprising straightforward. There are a number of big “tells,” such as echoing and amplifying obvious conspiracy theories, claiming that a single treatment cures multiple unrelated diseases, portraying an intervention as unrelentingly awful, and the like. Perhaps the single most important “tell” that identifies a physician or scientist is hostility towards a scientific consensus—or, often, hostility towards even the concept of scientific consensus, as I have described in detail multiple times.
Think of this as just another part of pre-bunking, the technique of communicating common (and fairly easily predicted) narratives of misinformation in order to “head it off at the pass,” so to speak. Part of pre-bunking is learning to recognize who is and is not a reliable expert source of information about a topic.
We also need to clean up our profession
It’s long been a theme on this blog that medicine as a profession and the governmental bodies that regulate it, such as state medical boards, have failed miserably at the task of policing medicine and protecting patients (and the public) from quacks and misinformation-spreading social media influencers. For nearly the entire 16 year history of this blog we have lamented how state medical boards are underfunded, understaffed, and lack the legal tools (and, in some cases, the will) to truly protect the public from rogue doctors. In particular, they are reluctant to judge when one of our own is a quack or to do anything that appears to be “policing speech,” never mind that antivax and quack speech nearly always goes hand-in-hand with unscientific practices that can be considered quackery. Nor is this a new problem, either.
I like to cite the case of Dr. Rashid Buttar, whose story began with an attempt by the North Carolina Board of Medical Examiners to discipline him for his use of chelation therapy to treat autism and cancer. During the proceedings, Dr. Buttar famously (to us, anyway) called the Board a “rabid dog.” Ultimately, however, instead of permanently revoking Dr. Buttar’s medical license, the Board gave him a slap on the wrist, telling him that he could no longer treat children or cancer patients, rather than stripping him of his license. This lead Dr. Buttar to become a “pioneer” in what has become an all-too-common and distressing tactic among quacks since COVID-19: Attacking state medical boards and any other medical authority trying to rein in quacks and misinformation. His was the template that COVID-19 disinformation spreading doctors followed. Dr. Buttar and his allies persuaded the state legislature to alter North Carolina state law to make it more quack-friendly by mobilizing public support to neuter the North Carolina Medical Board and prevent it from disciplining doctors who practice quackery associated with “integrative medicine.” Specifically, the 2008 law, passed due to the advocacy of the North Carolina Integrative Medical Society and Dr. Buttar, prevents the medical board from disciplining a physician for using “non-traditional” or “experimental treatments” unless it can prove they are ineffective or more harmful that prevailing treatments. Again, this was the template that became the norm during the pandemic and was used to attack state medical boards that tried to discipline physicians for promoting COVID-19 misinformation.
I realize that the politics have become very toxic, but some of this is a self-inflicted wound, due to the long history of what Dr. Val Jones dubbed—in 2008!—”shruggies,” doctors who recognize quackery among their peers but don’t do anything about it, simply shrugging their shoulders (hence the name). Moreover, if a large majority of doctors favored disciplining our own for spreading antivax misinformation, lies, and conspiracy theories, legislatures would listen. Instead, our profession is too concerned with protecting its financial status and its “freedom of speech,” the latter of which quacks appeal to in order to persuade doctors who might find them reprehensible to stay on the fence, at minimum. Until we as a profession recommit ourselves to dedication to science-based medicine, to countering quack narratives, and, most importantly, to cease tolerating the quacks (antivax or other) among our members, the problem will persist and likely continue to get worse.

Why Are Sensible Medicine Doctors Indifferent to the Nonsense Comments of Sensible Medicine Commentators? As Instructed, I Consider the Dualities of Interest and Motivating Biases.
Sensible Medicine doctors often utter the words “do an RCT”. So why are their readers so ignorant about RCTs?
Jonathan Howard on January 28, 2024
Link
“Conflicts of interest are an ongoing threat to medical practice.”
In their must-read article, Subscription Science: How Crowdfunding Has Become a Conflict of Interest, Drs. Benjamin Mazer, Michael Rose wrote that “Conflicts of interest are an ongoing threat to medical practice.” They further noted that:
Digital platforms such as Patreon, Substack, YouTube, and Twitter allow fans to offer recurring payments to healthcare professionals who produce opinion articles, explanatory videos, and podcasts… Substack, an online newsletter platform, is increasingly used by physicians to write medical commentary, with some newsletters reaching 10s of thousands of subscribers. Substack estimates that 5-10% of readers will upgrade to a paid tier, and paid subscriptions on the service cost a minimum of $5 per month. Although $5 sounds negligible, consider a newsletter with 10,000 total subscribers, 1000 of whom pay a $5 monthly fee. After subtracting Substack’s 10% cut, a doctor could expect $54,000 in annual payments. If physicians accrue 5000 backers, they can expect $270,000 in revenue. This is greater than the $265,000 average salary of primary care physicians in the US.
“Is a ‘model’ more believable than a double blinded, randomized, placebo-controlled study, with an N of 80,000?“
With this in mind, let’s revisit an article titled Not the Way to Pull Vaccines Back from a “Dangerous Tipping Point by Dr. Adam Cifu, a “medical conservative” and founder of Sensible Medicine, a Substack with 75,000 subscribers and 3 payment models.
Becoming a “Founding Member” for $250 per year
To his credit, Dr. Cifu wrote, “I am convinced of the safety of the COVID vaccine and know that the initial series saved millions of lives.” Dr. Cifu linked to a modeling study that concluded:
From December 2020 through November 2022, we estimate that the COVID-19 vaccination program in the U.S. prevented more than 18.5 million additional hospitalizations and 3.2 million additional deaths. Without vaccination, there would have been nearly 120 million more COVID-19 infections. The vaccination program also saved the U.S. $1.15 trillion (Credible Interval: $1.10 trillion–$1.19 trillion) (data not shown) in medical costs that would otherwise have been incurred.
Of course, we’ll never know exactly how many lives the COVID vaccines saved. We don’t have an Earth 2.0 where we can rerun the pandemic without them, and thus we need modeling studies to estimate their benefit. This situation is obviously not unique to vaccines. Modeling studies are needed to estimate the benefits of seat belts, for example.
Unsurprisingly, Sensible Medicine readers were unimpressed with this modeling study. They have been taught to reject all studies unless they are randomized controlled trials (RCTs)- “observational data has been used to support vaccines, but is plagued by confounding” – or they show potential vaccine harms. Indeed, one of the most liked comments on Dr. Cifu’s article said:
“know that the initial series saved millions of lives”.
The study you linked to is a model that claims:
“U.S. has administered more than 655 million doses”, “U.S. prevented more than 18.5 million additional hospitalizations and 3.2 million additional deaths.”
Let’s do a back of the envelope calculation. 655,000,000/3,200,000 = 204.6. That’s one life saved for every 205 doses. 655,000,000/18,500,000 = 35. That’s one less hospitalization for every 35 doses.
Add the two clinical trials together (Moderna, Pfizer). ~80,000 doses. 80,000/205 = 390,
80,000/35 = 2285. This predicts that we should have seen about 390 more deaths and 2285 more hospitalizations in the placebo group of the combined studies– but we saw no statistically significant reduction of death or hospitalization in these studies.
What was significant in the Pfizer study: -29 severe Covid events, +112 severe vaccine events. Looks like a net loss to me.
Is a ‘model’ more believable than a double blinded, randomized, placebo-controlled study, with an N of 80,000?
“The numbers are already cooked,” added another typical Sensible Medicine commentator in agreement.
“The primary analysis will be performed when approximately 151 cases have been observed in the study”.
Is this commentator correct? No, he is not.
The Sensible Medicine commentator fundamentally misunderstood how the COVID vaccine RCTs were structured. Perhaps if this person had been reading Science Based Medicine instead, they’d already know the COVID vaccine RCTs were designed to end when a small number of people contracted COVID. According to the Moderna protocol, “The primary analysis will be performed when approximately 151 cases have been observed in the study”. The Pfizer protocol set “a target of 164 primary-endpoint cases of confirmed Covid-19”. The Sensible Medicine commentator would have been sensible had all 80,000 people in the RCTs contracted COVID, but by design the trials concluded before the virus was allowed to harm more than several hundred people, 347 as it turned out.
The results were pretty impressive.
    • In the Pfizer RCT, 21,720 people received the vaccine and 21,728 received placebo. There were 162 COVID cases in the placebo group and 8 in the vaccine group. There were 10 cases of severe COVID-19, 9 of which were in the placebo group, 1 the vaccine group.
    • In the Moderna RCT, there were 15,210 participants in the vaccine and placebo groups. Symptomatic COVID occurred in 185 participants in the placebo group and in 11 participants in the vaccine group. Severe COVID occurred in 30 participants, with one fatality, all in the placebo group.
Incredibly, according to this typical Sensible Medicine commentator, these results looked “like a net loss” for the vaccine.
“Motivating biases need not be considered nefarious, only considered.”
I am not responsible for the comments on my writing and neither is Dr. Cifu. However, the people who take the time to respond to us reflect on us in some way, and our response to them, or our decision not to respond, reflects on us entirely. Though it happens rarely and I don’t see everything, if I spotted a ridiculous comment like that on one of my articles, I’d damn well correct it, assuming one of our astute readers had not done so already. I don’t want my writing to be a vehicle for people to spread anti-vaccine nonsense about RCTs. Part of being pro-RCT is correcting misinformation about actual RCTs, especially in spaces where one has some control and authority.
In contrast, Sensible Medicine doctors said nothing. They’ve said “do an RCT” countless times, but can’t be bothered to educate their readers about the RCTs that have actually been done. Given their purported admiration for RCTs, don’t you think they have an obligation to teach their misinformed readers about the basics of how the actual COVID vaccine-RCTs were designed and how they should be interpreted?
I do.
It’s revealing they haven’t done so already. As Dr. David Gorksi and I have discussed many times (here, here, here, here) these doctors performatively fetishize RCTs not to advance actual RCTs or to educate their readers about RCTs, but rather to stoke anger and foster mistrust. The ignorance of their commentators shows the predictable result.
Beyond this, Sensible Medicine readers must pay a fee to comment. Not only does this business model foster an echo-chamber of groupthink, it also creates dualities of interest and and motivating biases for Sensible Medicine doctors. After all, correcting commentators risks offending paying customers, even those who spread blatant falsities about precious, glorious, hallowed RCTs.
It’s entirely possible this financial incentive plays no role in refusal of of Sensible Medicine doctors to call out nonsense from their commentators, especially considering their own copious misinformation. However, I’m perfectly willing to follow the wise instructions they provided in their foundational paper, The Case for Being a Medical Conservative:
Dualities of interest must be considered in determining the quality of evidence…The medical conservative, therefore, is pragmatic about human nature and the prevailing business model of medical science. To wit, content experts, professional societies, or journal editors who too harshly criticize an industry product jeopardize future funding. Motivating biases need not be considered nefarious, only considered.
If you think I’m wrong for considering these dualities of interest and motivating biases, let me know. Everyone is welcome to comment here, and it won’t cost you a penny.

I Reject Your Medical Reality and I Substitute Death
Denying reality and substituting his own worked for Adam Savage. Not so much as an approach to COVID.
Mark CrisliponAugust 15, 2023
Link

I am, no surprise, a big fan of reality. I think I have a reasonable approximation of what reality is, mostly understood through the lens of the sciences. I learned early in my edumacation the key word is approximation. Always approaching a complete picture of life, the universe, and everything asymptotically. Zeno’s paradox is the best representation of to how I acquire understanding.
Not everyone is a fan of reality. I remember being amused back in the day by Adam Savage’s tag line of,”I reject your reality and substitute my own.” As a sci-fi guy who read a lot of alternative (complementary? integrative?) history, I thought it funny. But to a rather worrisome degree, it defines many people, except for the obvious (to me) fact that there are no other realities that can be slotted in at will.
I have often thought it should be reality-based medicine. In part, because the SCAM proponents use science, albeit, with a more Dr. Science kind of ineptness, to justify their practices, and in part it offers an easy opposite. What is the opposite of science-based medicine? The antonyms of science do not roll trippingly off the tongue, although I do kind of like witlessness-based medicine as an alternative to alternative medicine. But the opposite of reality? Fantasy-based, delusion-based, or fiction-based medicine sum the not-so-distinguished competition up so much better.
Denying reality has consequences. Few, rock climbers excepted, deny, say, the existence of gravity. Planes and bridges are not built using alternative ideas as to what constitutes gravity. The earth sucks, as an example, is not a workable construct for keeping a plane aloft.
Medicine seems particularly prone to people denying reality and substituting their own. It often doesn’t work out well. I saw many examples in my long and storied career. The first case was as a medical student doing my ER rotation. Guy came in weak with a low blood pressure and a very low hemoglobin (a measurement of how much red blood cells are in the circulation). A slow bleed, from a gastrointestinal cancer, was on the list of reasons, but taking off his shirt to listen to his lungs revealed a huge, mushroom-shaped melanoma that was oozing blood. He denied its existence. He was not the only patient I saw over the years who presented with huge, necrotic tumors whose presence were denied by the patient. Some were the size of dinner plates and the patient would say, nope, hadn’t noticed.
I have mentioned in the past the case that got me involved in SCAMs. I was called for a consult for a leg infection. Rather than an infection, it was a young female with a dead, rotting leg from a progressive sarcoma being very unsuccessfully treated by a naturopath. The tumor eroded that night into an artery and she died. ND. Not a Doctor.
I also saw many a patient who denied their AIDS diagnosis, refused care, progressed and died.
Note a pattern: reject medical reality and die.
There were no end of deaths from COVID in those who denied COVID, its prevention, appropriate treatment or the benefits of the vaccine. Reality substitution seemed to have a particular predilection for conservative talk show hosts
no less than five conservative radio talk show hosts who urged their audiences to avoid vaccines for COVID-19 have died from the virus.
Far be it for me to suggest any schadenfreude. Those COVID deaths from reality substitution were mostly anecdotes, not a careful evaluation of the epidemiology of the consequences of ignoring reality.
The epidemiology of COVID changed as the virus washed back and forth over the US. It started in the blue NW and took its time to reach the red South. The disease evolved and along with it our understanding of its spread, its treatment and prevention. So early on it was difficult to make hard-and-fast rules about who gets COVID and why, given all the flux and variation.
This was noted in early studies as COVID wandered from blue to red states.
From March 2020 to early June 2020, Republican-led states had lower COVID-19 incidence rates than Democratic-led states. On June 3, 2020, the association reversed, and Republican-led states had a higher incidence (risk ratio=1.10, 95% posterior interval=1.01, 1.18). This trend persisted through early December 2020. For death rates, Republican-led states had lower rates early in the pandemic but higher rates from July 4, 2020 (risk ratio=1.18, 95% posterior interval=1.02, 1.31) through mid-December 2020.
And while the death rates/100,000 were higher in states with Republican Governors, the comparisons were not at a steady state for the disease in the country.
Substituting reality by ignoring masking and social distancing before the vaccine was variable and direct comparisons of red or blue were problematic. The trend towards more reality substitution in red areas and subsequent suboptimal consequences appeared real and was plausible from basic infection control practice. Reality substitution is a bad idea in the hospital for infection control, there would be no reason to suspect the same principals would not apply in the real world.
So, when did we reach steady state with COVID? Got me. Sometime after the vaccine was released and after the entire country had seen the first wave would be one reasonable, or arbitrary, point. And that brings us to the most recent reality denial paper, Excess Death Rates for Republican and Democratic Registered Voters in Florida and Ohio During the COVID-19 Pandemic
Between January 1, 2018, and December 31, 2021, there were 538 159 individuals in Ohio and Florida who died at age 25 years or older in the study sample. The median age at death was 78 years (IQR, 71-89 years). Overall, the excess death rate for Republican voters was 2.8 percentage points, or 15%, higher than the excess death rate for Democratic voters (95% prediction interval [PI], 1.6-3.7 percentage points). After May 1, 2021, when vaccines were available to all adults, the excess death rate gap between Republican and Democratic voters widened from -0.9 percentage point (95% PI, -2.5 to 0.3 percentage points) to 7.7 percentage points (95% PI, 6.0-9.3 percentage points) in the adjusted analysis; the excess death rate among Republican voters was 43% higher than the excess death rate among Democratic voters. The gap in excess death rates between Republican and Democratic voters was larger in counties with lower vaccination rates and was primarily noted in voters residing in Ohio.
How widely applicable the results are is uncertain, but it is in agreement with the general trend that denying reality and substituting your own can be fatal, at least in medicine. It still worked well as a concept in MythBusters.
But consider the close election of 2024.
Arizona had 6400 COVID deaths as of November 2020 on their way to 30,000 as of this month.Biden won by 10,000 votes in 2020.
Georgia had 8500 deaths on the way to 35,000.Biden won by 12,000 votes.
Wisconsin had 3000 deaths on the way to 16000.Biden won by 20,000.
Republicans had a 43% higher death rate since the vaccine was introduced and a 15% higher death rate before the vaccine.
Ballpark calculations, and there are all sorts of ways to crunch the numbers depending on the breakdown of party affiliation, age, etc., suggests that reality denial may have helped prevent the re-election of Donald Trump. Odd to think about, but the Red Cult of Death’s reality substitution may have helped swing the election to Biden, what with the electoral college and all. Trump still would have lost the popular vote.
And for 2024 there will be more red dead than blue dead. Supporting policies that kill your constituents may be counterproductive to electoral victory. If a similarly close race happens in 2024, it may not be the dead who vote that make a difference, but the COVID dead who can no longer vote. Red Dead and no Redemption My son played that game during COVID; I particularly liked that the main character caught TB and died of it. Didn’t wear a mask or socially distance when it may have helped.
Someone with a more sophisticated understanding of the numbers should do the calculations across the US as of November 2022 and see how reality denial affected the election and might alter future elections.
In the meantime, I will continue to voice my support for reality in all its approximations, regardless of the electoral consequences.

Don’t call CAM “cost-effective” unless it’s actually effective
Before deciding CAM is cost-effective, it is important to determine if is just effective. That low bar has not been surmounted.
Scott Gavura on October 11, 2012
Link
Your health insurance plan probably covers anti-inflammatory drugs. But does it cover acupuncture treatments? Should it? Which health services deliver good value for money? Lest you think the debate is limited to the United States (which is an outlier when it comes to health spending), even countries with publicly-run healthcare systems are scrutinizing spending. Devoting dollars to one area (say, hospitals) is effectively a decision not to spend on something else, (perhaps public health programs). All systems, be they public or private, allocate funds in ways to spend money in the most efficient way possible. Thoughtful decisions require a consideration of both benefits and costs.
One of the consistent positions put forward by contributors to this blog is that all health interventions should be evaluated based on the same evidence standard. From this perspective, there is no distinct basket of products and services which are labelled “alternative”, “complementary” or more recently “integrative”. There are only treatments and interventions which have been evaluated to be effective, and those that have not. The idea that these two categories should both be considered valid approaches is a testament to promoters of complementary and alternative medicine (CAM), who, unable to meet the scientific standard, have argued (largely successfully) for different standards and special consideration — be it product regulation (e.g., supplements) or practitioner regulation.
Yet promoters of CAM seek the imprimatur of legitimacy conferred by the tools of science. And in an environment of economic restraint in health spending, they further recognize that showing economic value of CAM is important. Consequently they use the tools of economics to argue a perspective, rather than answer a question. And that’s the case with a recent paper I noticed was being touted by alternative medicine practitioners. Entitled, Are complementary therapies and integrative care cost-effective? A systematic review of economic evaluations, it attempts to summarize economic evaluations conducted on CAM treatments. Why a systematic review? One of the more effective tools for evaluating health outcomes, a systematic review seeks to analyze all published (and unpublished) information on a focused question, using a standardized, transparent approach to evidence analysis. When done well, systematic reviews can sift through thousands of clinical trials to answer focused questions in ways that are less biased than cherry-picking individual studies. The Cochrane Review’s systematic reviews form one of the more respected sources of objective information (with some caveats) on the efficacy of different health interventions. So there’s been interest in applying the techniques of systematic reviews to questions of economics, where both costs and effects must be measured. Economic evaluations at their core seek to measure the “bang for the buck” of different health interventions. The most accurate economic analyses are built into prospective clinical trials. These studies collect real-world costs and patient consequences, and then allow an accurate evaluation of value-for-money. These types of analyses are rare, however. Most economic evaluations involve modelling (a little to a lot) where health effects and related costs are estimated, to arrive at a calculation of value. Then there’s a discussion of whether that value calculation is “cost-effective”. It’s little wonder that many health professionals look suspiciously at economic analyses: the models are complicated and involve so many variables with subjective inputs that it can be difficult to sort out what the real effects are. Not surprisingly, most economic analyses suggest treatments are cost-effective. Before diving into the study, let’s consider the approach:
What do you mean, cost-effective?
Whenever you hear any treatment described as “cost-effective”, you should immediately ask, “Cost-effective compared to what?” Cost-effectiveness evaluations consider incremental costs and incremental benefits, and cannot be considered in isolation:
(Source: www.acponline.org)
The “CE ratio” is the effective additional cost for the additional benefit offered by a new strategy or treatment. If the new strategy provides better outcomes at the same or lesser cost as the current practice, then the new strategy is said to be “dominant” — that is, it should be accepted immediately. If the new strategy provides worse outcomes at a higher cost, then current practice is said to dominate — the new strategy should be abandoned. Few studies are this simple, however. Most published economic analyses tend to show interventions that claim better efficacy at a higher cost. (For an example, see the economic evaluation of virtually every new drug launched in the past 30 years.) If the CE ratio is low, the treatment is said to be “cost-effective” — there’s a lot of benefit for a modest additional cost. If the incremental cost is high, relative to the incremental benefit, the CE ratio is high, and the new strategy is deemed “cost-ineffective”. By now it should be clear that what’s cost-effective is ultimately a judgement call. What’s felt to be cost-effective might be influenced by how much money you have to spend, or how you prioritize your spending.
Given the number of variables in an economic analysis, the opportunity for bias is high. It all depends on the design. Some of the major considerations include:
What is the comparator? All cost-effectiveness evaluations compares one intervention to another. Substituting homeopathy for antibiotics for the treatment of colds may be a cost-effective intervention. After all both are ineffective, and homeopathy may be less expensive than antibiotics. What’s further, there is the risk of harm from adverse events due to drug, while homeopathy is inert, without medicinal ingredients. Does that mean homeopathy is “cost-effective” for colds? Not quite. Giving “no treatment” for a cold is superior to (or “dominates”) both antibiotics and homeopathy, as it is both safe and free of cost. If the comparator is inappropriate, and any derived cost-effectiveness estimate is irrelevant.
Just how effective is the intervention? In general, effective interventions tend to be cost-effective. Some are even cost saving. Vaccines are one example. Interventions, science-based or otherwise, that offer modest and very incremental benefits, may be deemed a cost-ineffective use of resources. Screening programs, for example, can be clinically effective, but tremendously cost-ineffective when you factor in harms of false-positives. Again, the best evidence for efficacy usually comes from prospective, randomized trials. Evidence can be derived from other settings, and there may be very good reasons why randomized data is unavailable (once again, vaccines are an example of this). However, this needs to be factored into the evaluation when considering overall effects, and the risks of bias. So when we’re looking at interventions, we need to consider not only the trial from which the economic evidence is derived, but the totality of evidence for that intervention. There are multiple studies that suggest that acupuncture has clinically meaningful effects. But the best and most comprehensive studies repeatedly conclude that acupuncture’s effects are placebo effects. If there is any doubt about the clinical effect of the intervention, then the economic analysis must reflect this. Otherwise, the value-for-money evaluation will be hopelessly biased. Recall the study of acupuncture for asthma that observed patient-reported improvements, but no objective improvements in control. Depending on what endpoints you base your economic model on, your conclusion will be very different.
How realistic is this intervention? Like any other clinical trial, data from research must be translated into the real world. If the interventions can be offered in the real world in a way that’s consistent with the trial, and the real-world patients are similar to those studied, we’d generally agree that a particular study is generalizable. As we move further away from the type of intervention studied and the types of patients that participated, the relevance of the clinical study to the real world becomes less clear. Patients in clinical trials behave differently than those in regular, everyday treatments. When we move from highly controlled trials to more pragmatic studies, we generally expect treatment effects to decline. This isn’t the case with CAM — rigorous studies show a lack of effect, while more “pragmatic” trials, studying real world use, are more likely to report benefit. While the treatment setting may be more realistic, the ability to distinguish objective effects from subjective effects may be more difficult.
What is the type of analysis? Different models answer different questions. Cost-effectiveness analyses compare the costs and effects of different treatments, when you can measure the effect objectively. Cost-benefit analyses are used to answer questions about overall benefits in a population, such as public-health interventions. Cost-utility analyses require measurement of effects that can estimate “quality-adjusted life years” or QALYs.
These are the big four, but there are several other areas that need to be scrutinized in an evaluation, including the accuracy of the measurement tools used, the assignment of accurate costs to different outcomes, and the overall sensitivity of the model to small changes in estimates. The key point to remember is that with so many knobs and buttons, any economic analysis can make a health treatment look good — especially if the clinical evidence it’s based on is favorable.
Given the variables that need to be examined, economic analyses don’t lend themselves systematic reviews, which are designed to synthesize all relevant information to answer a focused clinical question. And that seems to be the case with the paper in question. These authors attempt to use the framework of a systematic review to examine the economic evidence supporting CAM treatments. The lead author is Patricia Herman, a behavioral scientist at the RAND corporation who is also a naturopath. The second author is Beth L. Poindexter, also a naturopath, with no other publications cited in PubMed. Claudia Witt is a physician who holds multiple academic positions in complementary medicine and alternative medicine. She has numerous publications in her CV including studies of homeopathy for chronic illness. The final author is Dr. David Eisenberg, a medical doctor who is a proponent of CAM and naturopathy in particular. None of this disqualifies the authors or their economic analysis of CAM. However, it does raise concerns about the potential for bias to be present in the analysis.
The review attempts to identify all published economic evaluations of CAM, (which they call CIM, for complementary and integrative medicine). The authors point out that CAM is (not surprisingly) not funded by insurance plans:
In not being part of conventional medicine, individual complementary therapies and emerging models of integrative medicine (ie, coordinated access to both conventional and complementary care)—collectively termed as complementary and integrative medicine (CIM)—are often excluded in financial mechanisms commonly available for conventional medicine,2 and are rarely included in the range of options considered in the formation of healthcare policy. The availability of economic data could improve the consideration and appropriate inclusion of CIM in strategies to lower overall healthcare costs. In addition, economic outcomes are relevant to the licensure and scope of practice of practitioners, industry investment decisions (eg, the business case for integrative medicine), consumers and future research efforts (ie, through identifying decision-critical parameters for additional research10).
This is a textbook example of begging the question, assuming that CAM is being excluded from strategies to lower healthcare costs, without explaining why these treatments are not considered “conventional medicine,” and without demonstrating that these treatments actually lower overall healthcare costs. So the authors stated objective is to catalog the economic evidence that does exist.
The authors admit (and I agree) that search strategies for CAM in the published medical literature are difficult. It’s complicated in part by the industry’s decision to keep redefining itself — so what was labelled “alternative” is now “integrated” or “collaborative”. They exclude all treatments considered to be “standard treatment” or “delivered by conventionally credentialed medical personnel”. That is, they attempt to identify treatments which are either not proven to work, or proven not to work. The net result is 204 papers published between 2001 through 2010. The reviews span the CAM spectrum:
    • 45 were manipulative therapies (e.g., chiropractic) for a range of conditions
    • 41 were acupuncture treatments
    • 38 were studies of “natural” products
    • 27 were for forms of “mind-body” medicine
    • 24 were for homeopathy
    • 18 were for general CAM interventions
    • 25 were for other CAM therapies, including “aromatherapy, healing touch, Tai Chi, Alexander technique, spa therapy, music therapy, electrodermal screening, clinical holistic medicine, naturopathic medicine, anthroposophic medicine, water-only fasting, Ornish Program for Reversing Heart Disease, use of a corset and use of a traditional mental health practitioner.”
Unfortunately, in no part of the analysis is the veracity of the underlying clinical data questioned. The authors applied different economic analysis quality checklists, narrowing the list to 31 studies deemed “high quality”. And here’s where the weakness of the underlying evidence is made clear. There appear few if any papers based on studies that actually blinded patients. Many are studying the “adjunctive” or add-on addition of CAM such as acupuncture to usual care, and concluding that interventions are either cost-saving or offer good value for money. Some of the studies included are economic models only — not based on actual data collected in trials, but utilizing simulated models to estimate costs and effects. And some of the studies deemed CAM don’t appear to be CAM at all- one paper estimates cost-effectiveness of a clinical trial that would evaluate vitamin K1 versus alendronate for fracture prevention: neither an evaluation of CAM nor an evaluation of its underlying cost-effectiveness. Another models outlines the cost-savings of Tai Chi programs in a nursing home population, based on the premise that tai chi demonstrably reduces the risk of falls. Looking at the manipulative treatments, some were delivered by chiropractors, and others by physiotherapists or general practitioners. Given the variety of treatments and approaches included in the final list, it’s not possible to draw any conclusions about the individual or overall effectiveness of CAM. But that doesn’t stop the authors from trying:
There are several implications of this study for policy makers, clinicians and future researchers. First, there is a large and growing literature of quality economic evaluations in CIM. However, although indexing is improving in databases, finding these studies can require going beyond simple CIM-related search terms. Second, the results of the higher-quality studies indicate a number of highly cost-effective, and even cost saving, CIM therapies. Almost 30% of the 56 cost-effectiveness, cost-utility and cost-benefit comparisons shown in table 4 (18% of the CUA comparisons) were cost saving. Compare this to 9% of 1433 CUA comparisons found to be cost saving in a large review of economic evaluations across all medicine.
Here the authors make a direct comparison between CAM and conventional medicine, implying that CAM may save more than medicine — a comparison that can’t be substantiated based on the multiple weaknesses in the results that report. Contrast their finding with that of White and Ernst, in their 2000 systematic review of CAM economic analyses, make a more cautious and pragmatic conclusion about the economic attractiveness of CAM:
Spinal manipulative therapy for back pain may offer cost savings to society, but it does not save money for the purchaser. There is a paucity of rigorous studies that could provide conclusive evidence of differences in costs and outcomes between other complementary therapies and orthodox medicine. The evidence from methodologically flawed studies is contradicted by more rigorous studies, and there is a need for high quality investigations of the costs and benefits of complementary medicine.
White and Ernst’s conclusion is equally relevant to the current systematic review. When it comes to evaluating cost-effectiveness, we first need to establish that a treatment is actually effective. Without this, there’s no point in proceeding to looking at costs. The authors mention this indirectly — but not to determine if these interventions actually work, but to validate the results (and seemingly the conclusions) that have already been drawn about their effectiveness:
Given the substantial number of economic evaluations of CIM found in this comprehensive review, even though it can always be said that more studies are needed, what is actually needed are better-quality studies—both in terms of better study quality (to increase the validity of the results for its targeted population and setting) and better transferability (to increase the usefulness of these results to other decision makers in other settings).
Conclusion: CAM is not cost effective
The highest quality evidence, and most rigorous clinical trials, have failed to identify complementary and alternative medicine interventions that offer meaningful clinical benefits. Lower quality trials are more likely to report positive findings, and economic analyses based on these studies should be expected to be biased towards findings of cost-effectiveness. Economic analyses that conclude treatments are cost effective are more likely to be published. The result is an economic house of cards. Given the variety of treatments, models and endpoints, systematic reviews are not useful tools to evaluate the overall cost-effectiveness of a disparate group of treatments. However, they may be useful in summarizing the types of studies that exist. Given the results of this systematic review, there is good evidence to suggest that economic analyses of CAM interventions do not generate useful or accurate estimates of cost-effectiveness.
Reference
Herman, P.M., Poindexter, B.L., Witt, C.M. & Eisenberg, D.M. (2012). Are complementary therapies and integrative care cost-effective? A systematic review of economic evaluations, BMJ Open, 2 (5) e001046. DOI: 10.1136/bmjopen-2012-001046

The risks associated with alternative medicine
In a new paper, the types and severity of harms from different types of alternative medicine are described.
Scott Gavura on June 24, 2021
Link 
Alternative medicine is big business. While it may have the image of small companies, personalized treatments, and independent practitioners, it is a multi-billion dollar industry that is growing rapidly. In contrast with our definition of science-based medicine, alternative medicine is arguably harder to define, as it encompasses a variety of treatments, practices and therapies that range from unproven to disproven, and includes not only drugs (herbal or otherwise), physical interventions, and even spiritual/metaphysical practices. What all of these practices have in common is that that they are not accepted as part of conventional medical practice, which is grounded in a standard of scientific evidence. While the practices are disparate, alternative medicine is generally seen as safer, even “gentler”, than conventional medicine.
Despite perceptions, serious harms from alternative medicine have been documented in the medical literature and in this blog. Beyond the obvious direct harms of a toxic therapy (e.g., treating children with autism with bleach), harms can also be caused by avoiding effective therapies in favour of ineffective ones. They reporting of harms from alternative medicine is believed to be less common than conventional medicine mainly because of the robust and mandatory (in places) reporting built into delivery systems for conventional medical care. In contrast most alternative medicine is provided privately, and regulatory oversight of most alternative medicine products and practices tends to be lighter (e.g., the licensing frameworks for supplements vs. over-the-counter drugs). Perceptions may also feed this reality, as consumers that believe these treatments are harmless may not attribute adverse events to alternative medicines/practices.
Because harms are poorly understood, a group of researchers led by Dr. Bernie Garrett recently undertook a process to define the types of risks associated with complementary and alternative medicine (CAM) and to classify them into categories. The paper, entitled “A taxonomy of risk-associated alternative health practices: A Delphi study” was published in May 2021 and is open access.
The researchers used the Delphi method with a group of experts to develop a consensus document. Briefly, the Delphi method involves sequential group surveys with aggregated responses provided back to the group with the objective of finding consensus. They established a multi-disciplinary group of participants, including four physicians, four nurses, three pharmacists, three physiotherapists, two social workers, two lawyers (with expertise in harm and injury), an epidemiologist, a naturopath, and a chiropractor. During the course of the work, four individuals withdrew for different reasons (1xMD, 1xRN, 1xPT, 1xSW) leaving 17 in the panel.
The panel started with a literature review that sought to identify articles, legal cases and media reports that involved harms of CAM. Search terms were informed by the NCCIH categorization, which David Gorski blogged about some time ago:
Search terms were combined with searches for reports of “harms, risks deaths, damage, injury” and “adverse events” resulting in over 1,700 articles. This formed the basis of a list that was developed and sent to the panel for comment. Panelists were asked for feedback on their agreement with the developed list, the categorization used, and the level of risk associated with each item. Five sequential rounds of consultation and analysis occurred. The types of harms were simplified into the following framework:
As one would expect with a group that includes a naturopath, a chiropractor, and an array of health professionals, arriving at consensus on harms and risks of CAM was described as “challenging” as the group was noted to have “diverse opinions”.
Given the absence of systematically-collected data on harms, it was not possible to quantify risks in terms of probability. Three categories (high/moderate/lower) of risk were created based on significance of the harm itself, the existence of verified case examples, and the potential consequences vs. value of treatment:
    • Higher: exposes person seeking therapy (or others) to risk of serious and/or permanent physical or psychological harm, or death. Occurrences may be rare but the consequences are considered to outweigh any asserted potential value.
    • Moderate: exposes person seeking therapy (or others) to risk of (potentially reversible) significant physical, psychological, or economic harm. Occurrences may be rare but the potential consequences may outweigh any asserted potential value.
    • Lower: exposes person seeking therapy (or others) to a short-term risk of physical, psychological, or economic harm. Occurrences may be rare but indicate in some instances using it may be harmful.
A new definition of CAM
The authors note that NCCIH defines CAM as “a group of diverse medical and health care systems, practices, and products that are not considered to be part of conventional or allopathic medicine” and “a non-mainstream practice…used together with conventional medicine”. They note that the NCCIH definition depends on the cultural frame of reference – especially the term “complementary” which implies a therapy that is adjunctive to conventional care. The panel developed their own functional definition:
The range of therapeutics that largely originate from traditions and theories distinct from contemporary biomedical science, and which claim mechanisms of action outside of those currently accepted by scientific and biomedical consensus.
General harms
Many harms are not specific to any particular form of CAM. The following general harms were identified, with the level of harm in brackets.
    • Using alternative health care instead of the existing biomedical standard of care for medically treatable conditions (higher)
    • Using alternative therapeutics which are new and where side-effects are unknown (higher)
    • Using alternative therapeutics alongside existing medical treatments without informing the medical provider (higher)
    • Using alternative health treatments for diagnoses not currently recognized as biomedical illnesses or misdiagnosed (do not meet biomedical diagnostic criteria), for example, candida overgrowth, adrenal fatigue, chronic Lyme disease, etc. (moderate)
    • Utilizing alternative health care for the treatment of medical conditions based on misinformation in deceptive advertising/marketing (moderate)
    • Taking part in an alternative health research that has not been approved by an independent research ethics board (lower)
    • Impact of the financial costs associated with the use of alternative practitioners/therapeutics not covered under public health care provision (lower)
    • Societal economic impact associated with the use of alternative health care provision when covered by third parties (lower)
Alternative belief system harms
Some harms are unique to specific alternative medicine practices. The panel identified the following harms:
    • Traditional Chinese Medicine (TCM)
        ◦ Toxicity associated with specific TCM products – some products associated with cardiovascular, liver, kidney, and other harms. Some contain high levels of lead or mercury (higher)
        ◦ Injuries from cupping – bruising and burns which can be severe (moderate)
        ◦ Injuries from acupuncture – infection, trauma, collapsed lungs, and nerve damage (higher/moderate)
        ◦ Injuries from acupuncture with moxibustion – as above, plus burns (moderate)
        ◦ Injuries from acupuncture in vulnerable populations – increase risk of trauma/harm (lower)
    • Naturopathic and homeopathic practices
        ◦ Adverse events with naturopathic intravenous therapies – including infections, contamination, use of illegal or compounded products (higher)
        ◦ Injuries from colonic irrigation therapies with water, coffee and other substances – including infections, tears and intestinal perforation (higher)
        ◦ Provision of anti-vaccination advice – common with naturopathic and homeopathic belief systems (higher)
    • Ayurvedic practices
        ◦ Toxicity associated with specific remedies – numerous reports of lead poisoning, many medicines contain at least one metal (higher)
    • Religious health advice and faith healing
        ◦ Negative consequences of accepting advice that conflicts with medical advice (higher)
Physical manipulative alternative health care activities
The third major class of harms involved manual therapies: chiropractic, osteopathy, and massage:
    • Chiropractic
        ◦ Injuries from spinal manipulative therapy – including vascular dissection, stroke, and subdural hematoma; neurological damage; fractures. This includes an array of SMT activities. (varies from higher to lower)
        ◦ Injuries from spinal manipulative therapy in vulnerable populations or taking specific medications – including chiropractic in infants and children, the elderly, those with clotting disorders, etc. (higher to lower)
        ◦ Adoption of anti-vaccination advice – common in chiropractic, sometimes chiropractic is recommended instead of vaccination (higher)
    • Massage therapy
        ◦ Injuries with massage therapy in the elderly – musculoskeletal injuries (lower)
    • Osteopathic
        ◦ Injuries with prolotherapy injections – nerve damage when injections are performed near peripheral nerves (moderate)
Herbal and nutritional supplement therapies
    • The problems with herbal remedies and dietary supplements have been well documented at this blog, and are similar to concerns about TCM products:
    • Toxicity with specific remedies/supplements that contain metals – owing to a lack of quality control (higher)
    • Toxicity with specific remedies/supplements that are adulterated with drugs – owing again to a lack of quality control (higher)
    • Adverse effects of specific remedies/supplements – these may be unlisted on the product (higher)
    • Adverse effects of remedies/supplements for weight loss – reports of kidney and liver damage (higher)
    • Adverse effects of remedies/supplements used in vulnerable populations – such as use in pregnancy or while breastfeeding, or in the elderly (higher to moderate)
A new approach to understand the harms of CAM
Whether or not you are a user of CAM, everyone should be in favor of a safer system, where harms are minimized, documented, and accepted/understood by consumers. This is particularly important when the effectiveness is questionable or unclear, which is the case for the vast majority of CAM. These products and treatments are not innocuous and some are associated with reports of significant harm – even death. In this new paper, a diverse group of professionals reached a consensus on the types and severity of harms that can arise from complementary and alternative medicine. Further work could potentially build on this framework, in time providing providers and consumers alike with a better understanding of the risk and consequences of these therapies and treatments.

Patients Still Respect Evidence
Steven Novella on October 3, 2012
Link
A recent survey about patient attitudes and desires with regard to health care demonstrate that respect for scientific evidence is still the dominant factor in preferring treatments. (Full study) This is good news, although the numbers could be better.
Researchers asked subjects what factors were important in determining which treatments they would prefer, the scientific evidence, the experience of the clinician, or their own personal preferences. Not surprisingly, most subjects wanted it all, agreeing that all three are important. Scientific evidence, however, scored the highest with 71% rating it as very important (and over 90% as important or very important). Clinical expertise had 61% strongly supported and personal preference, 57%.
Further, patients wanted their doctors to talk to them about the evidence. The phrase they felt had the most impact on their decision to accept a treatment was, “What is proven to work best.”
All of this matches my personal experience as a clinician. At least for the self-selective population of patients who seek out a university physician, patients tend to find recommendations based upon published evidence compelling, and greatly appreciate when I take the time to tell them about the evidence, even if it goes against their initial interests.
Patients, for example, frequently ask me what I think about acupuncture, a particular supplement, or some other “alternative” treatment for their condition. I tell them – without waffling or watering down my opinion. I don’t editorialize or express judgmentalism because that is inappropriate in a therapeutic context, but I tell them my understanding of the published scientific evidence and the plausibility of the treatment, and then give them my bottom-line recommendation. I do this as if I were talking about any treatment option. Even when patients are starting from a very different opinion about the treatment, they appreciate the fact that I have taken the time to look into the research and to communicate that to them.
This is exactly why the “shruggie” approach to unconventional treatments is counterproductive and a huge disservice to patients. Saying, “I don’t know” when asked about an implausible and ineffective treatment, or even giving a dismissive response, is not going to be convincing to patients.
The survey also found that taking the time to communicate to patients about the scientific evidence greatly improves their satisfaction with their health care – another way in which the dismissive approach has a negative impact.
Despite an ongoing effort by some promoting treatments that are not science-based, there is still huge public support for science in general and science-based treatments. That is why proponents of dubious treatments are so desperate for the trappings of scientific legitimacy, even to the point of rewriting the rules of science to accommodate their preferred treatments. This is turn leads to the need to have watchdog groups defending the role and integrity of science in the practice of medicine.
The bottom line of this survey is that patients respect evidence and want their physicians to talk to them about it, although only 35% reported that their doctors take the time to explain the latest medical evidence. In order to improve this situation physicians need to understand the evidence not only for treatments they use, but also for treatments their patients are likely to ask them about, including CAM therapies.
I need to point out that, like all surveys, this data needs to be taken with a grain of salt. How questions are asked has a significant impact on how they are answered. Also, it would have been interesting to have control questions, asking patients how they feel about treatments that are not based on evidence, or that are based upon tradition or treatments for which a possible mechanism is not understood by science. Would patients accept a treatment that is lacking evidence or even if the evidence indicates it does not work if it were supported by anecdotal evidence? This survey, it can be argued, had a significant bias toward scientific evidence.
Regardless of the limitations, it does draw attention to the need for physicians to engage with individual patients, and, I would argue, the broader society, about the role, methods, and findings of science in medicine.

Americans still (mostly) trust scientists and doctors, but there are some troubling warning signs
Given all the denial of the science behind vaccines, GMOs, evolution, and climate science, you might think that Americans in general distrust scientists and physicians. It’s actually not true. Trust in scientists and doctors remains high, but there are still areas where mistrust of scientists is a significant problem. What can be done?
David Gorski on August 12, 2019
Link

One of the things I write a lot about, both here and at my not-so-super-secret other blog is science denial. This denial of science takes many forms, including denial of the conclusions of climate science that the earth is warming due to human activity, denial of evolution, denial of the science showing that vaccines are safe and effective and do not cause autism or myriad other conditions for which antivaxers blame them, and many more. Here, most commonly, we tend to write about denial of vaccine science and the antivaccine movement, although other relevant topics include HIV/AIDS denialism (the denial that HIV causes AIDS), denial by a number of cancer quacks that chemotherapy works, and a more general denial of scientific medicine by a number of practitioners of “natural medicine.” This science denialism is virtually always coupled with one or more conspiracy theories. The one I discuss the most is the antivaccine conspiracy theory that claims that the CDC “knows” that vaccines cause autism but has covered up and suppressed the evidence for it. Cancer quacks like to claim that a cure for cancer exists, but that big pharma, oncologists, and the government are keeping it from the people and suppressing knowledge of it in order to maintain their income and profits. Given this level of suspicion, you’d think that the public harbors a high degree of mistrust for scientists and physicians, A new poll released earlier this month by the Pew Research Group shows that that’s (mostly) not true:
In an era when science and politics often appear to collide, public confidence in scientists is on the upswing, and six-in-ten Americans say scientists should play an active role in policy debates about scientific issues, according to a new Pew Research Center survey.
The survey finds public confidence in scientists on par with confidence in the military. It also exceeds the levels of public confidence in other groups and institutions, including the media, business leaders and elected officials.
If you look at the accompanying graph of one of the overall conclusions of the poll, the percentage of Americans who say they have a great deal or fair amount of confidence in scientists to act in the best interests of the public, you’ll see that 86% of Americans trust scientists to act in the best interests of the public, up from 76% in 2016, with 35% saying they have a “great deal” of confidence, up from 21% in 2016. For medical scientists, the percentage is 87%, up so slightly over 84% in 2016 as to be within the margin of error. By way of comparison, business leaders only total 46% positive responses, with single digit percentages of respondents answering that they have a “great deal” of confidence in business leaders to act in the public interest. Elected officials fare even worse, with only 35% positive responses and even lower single digit percentages answering that they have a “great deal” of confidence in them, although, oddly enough, the percentage was only 27% in 2016. (Given the state of our politics in 2019, I’m finding it odd that more people express more confidence in our politicians to do right by the public, even if the overall confidence level remains abysmally low.)
When Pew gets into the weeds, there are a number of interesting findings regarding who has confidence in scientists and which kinds of scientists are the most trusted. Indeed, it’s rather akin to looking at a state’s overall vaccination level, seeing that it’s high, and dismissing concerns about low vaccine uptake even though there are pockets of low vaccine uptake in the state that could predispose to outbreaks. Trust in scientists and physicians might be high overall, but there are nonetheless major areas of concern. I’m going to concentrate mainly on the Pew Research Group’s findings with respect to public trust of medical and nutrition science, because, well, the name of this blog is Science-Based Medicine.
Americans trust practitioners more than scientists, but distrust nutrition science
The Pew survey asked respondents whether scientists in six different specialties can be counted on to act with competence, present their recommendations or research findings accurately, and care about the public’s best interests—or, in the case of physicians, their patients’ best interests. In addition, Pew asked about potential sources of mistrust, including issues of transparency and accountability for mistakes or misconduct. One conclusion of this poll is that Americans tend to trust “science practitioners” (such as physicians and dietitians) more than they trust science or medical researchers. For example, a majority of respondents (54%) say that dietitians do a “good job providing recommendations about healthy eating all or most of the time,” compared with only 28% who say nutrition scientists do a good job conducting research all or most of the time. Personally, I wasn’t surprised by these figures, given how the media report scientific and epidemiological studies of nutrition, particularly the narrative of, “Food X prevents cancer” followed later by a report of a study that finds that Food X either doesn’t prevent cancer or causes cancer. Indeed, given the uncertainty in a lot of nutritional research, coupled with the sensationalistic way the media like to report on nutritional research, it’s not at all surprising that there is so little trust in the research. What I do find surprising is that Pew would spin a finding that only a little more than half of people trust dietitians to be a positive finding, or that these are positive:
In addition, 47% say dietitians provide fair and accurate information about their recommendations all or most of the time, compared with 24% for nutrition scientists discussing their research. Six-in-ten Americans (60%) think dietitians care about the best interests of their patients all or most of the time, while about half as many (29%) believe that about nutrition researchers when it comes to concern for the public.
Again, those are pretty low numbers. The numbers are even lower for environmental health scientists and practitioners:
In contrast, public levels of trust in environmental health specialists and environmental research scientists are roughly the same. For instance, 39% of U.S. adults say environmental health specialists do a good job versus 40% for researchers, and 35% say each provides fair and accurate information all or most of the time.
Those are horrible numbers and could easily be one reason for how easily denialism of environmental science can take root.
When it comes to physicians, the numbers are considerably better, but still not the greatest. Although 74% of the public view physicians mostly positively, there are still problems:
Similarly, the public tends to view medical doctors more positively than medical researchers when it comes to their concern for the public’s interests and providing trustworthy information. For example, 57% of Americans say doctors care about the best interests of their patients all or most of the time, compared with 35% for medical researchers. About half the public (48%) believes that medical doctors provide fair and accurate treatment information all or most of the time, compared with 32% who say this about medical researchers in discussing their findings.
Less than three-fifths of the public think that doctors care about the best interests of their patients all or most of the time? Only 49% of people believe that doctors do a good job (another finding of the survey)? We have a major trust problem here.
Sources of mistrust and trust in scientists
What could be the reasons for this lack of trust? Pew asked respondents about conflicts of interest, transparency, and scientific misconduct. Half of the respondents say that misconduct is a big problem (15% very big and 35% moderately big) among medical doctors, with 48% answering the same (14% very big and 34% moderately big) about medical researchers. Among the other scientific specialties examined, the percentages ranged from 24% for dietitians to 42% for environmental research scientists. Again, we have a trust problem here.
One set of potential reasons:
No more than 19% say that scientists across these six specialties are transparent in revealing potential conflicts of interest with industry all or most of the time. A larger share – ranging from 27% to 37% – believes scientists are transparent only a little or none of the time. Similarly, fewer than two-in-ten Americans say that scientists admit and take responsibility for their mistakes all or most of the time.
If you want to know why antivaxers are so quick to invoke the “pharma shill gambit” when attacking scientists, physicians, and vaccine advocates, look no further. Yes, it fits their conspiratorial mindset and they invoke this gambit so often as to make it easy to joke about it, but it’s also a criticism that has a chance of gaining traction, given how many people view scientists as insufficiently transparent about their financial conflicts of interest.
There are also large numbers of people who believe that scientists don’t face sufficient consequences for misconduct. Depending upon the specialty, up to half of US adults say that scientists and physicians face serious consequences for misconduct “only a little” or “none of the time”. Specifically, 53% say that about nutrition researchers; about dietitians, 47%; about environmental researchers, 48%; about medical researchers, 45%; and about environmental health researchers, 42%. Oddly enough, only 30% say that medical doctors rarely face consequences for professional misbehavior.
We know what fuels mistrust, but what fuels trust in scientists? Pew found that people’s level of familiarity with scientists and their level of factual knowledge about science “can be consequential for public trust in scientists”. To sum it up, people with higher levels of factual knowledge about science tend to hold more positive and trusting views of scientists. However, the effect of these factors in promoting trust in scientists tends to be more limited than the effect of transparency, conflicts of interest, and admitting mistakes are in driving down trust in science. For example, among physicians, the percentage of respondents who say that doctors provide accurate information about their recommendations all or most of the time is 44% in those who know a little bit about doctors and 56% among those who know a lot. The percentage given the same answer to the question is 34% among those with little science knowledge and 58% among those with a lot of science knowledge. For medical science researchers, the percentage giving the same answer to the question is 17% in those who know a little bit about medical researchers and 53% among those who know a lot about them. Those with low science knowledge only give those answers 22% of the time, while those with high science knowledge respond that way 41% of the time.
Another unsurprising finding is that race plays a role in trust in science. For example, a large majority of black Americans (71%) and Hispanics (63%) respond that misconduct by medical doctors is a very/moderately big problem. Compare this to the finding that only 43% of whites give the same response to the question, and you can see that minorities distrust physicians way more than Caucasians do. The same is true regarding people’s opinion of medical researchers. A larger percentage of blacks (59%) and Hispanics (60%) say that misconduct by medical research scientists is a very big or moderately big problem, while only 42% of whites do. It’s not news to physicians that minorities, especially blacks, are much less trusting of doctors. Given the history of medicine in the US, racial inequities in medical care, and the ongoing relative paucity of black physicians, increased suspicion is not unreasonable. Even though medicine has been trying to alleviate medical disparities based on race, recruit more minorities into medicine, and reduce implicit bias among physicians, that history is hard to overcome.
Partisan political differences in public trust of scientists
No discussion of sources of mistrust of medicine and science would be complete without the examination of political effects. I’ve noted in the past, for instance, that, although antivaccine views are roughly equally prevalent on the right and on the left, right now the most prominent and politically active antivaxers tend to be on the right. So it was of great interest to me to see how partisan political beliefs influence trust in scientists.
Not surprisingly, there were wide differences in trust noted, but those differences tended to focus in certain specialties. For example, comparing the responses of people who are Republican or independents who lean Republican with those of people who are Democrats or independents who lean Democratic, Pew found huge differences in trust of environmental practitioners and scientists. (For purposes of brevity, I will just refer to each as Republicans or Democrats, with independents leaning towards each party lumped in with party members.) Way fewer Republican respondents (40%) reported having a mostly positive view of environmental research scientists than Democratic respondents (70%). Similarly, more Democrats (73%) than Republicans (46%) have a mostly positive review of environmental health specialists. Differences persisted for nutritional scientists, with 57% of Democrats and only 43% of Republicans trusting nutrition research scientists, the equivalent split being 63%/58% for dietitians. When it came to physicians and medical researchers, the differences collapsed to near or at the margin of error, with the Democrat/Republican split on this question being 77%/73% for physicians and 70%/67% for medical research scientists.
The obvious conclusion here is that areas where there is more partisan disagreement about policy show a wider partisan split in favorability rating. This split is even more pronounced when it comes to public perception that a given scientific discipline provides fair and accurate information most of the time. When it comes to the findings of environmental research scientists, only 19% of Republicans trust them to do this, while 47% of Democrats do; for environmental health specialists, the Republican/Democratic split is 25%/43%. On the question of whether environmental scientists care about the best interests of the public all or most of the time, the split was 22%/50%, and for the question of whether they do a good job conducting research all or most of the time, the split was 26%/51%.
There are a couple of remarkable things about these results. First, although I wasn’t surprised that Republicans would be very distrustful of environmental science, given Republican alignment with the interests of business, that climate science is a huge part of environmental science, and that climate science denialism has over the last couple of decades become baked into GOP ideology, I was amazed that the percentages saying they trust environmental scientists were so low among Democrats, less than 50%. One must wonder if the decades-long campaign to demonize climate scientists has had an effect on the public overall. Second, there’s a devastating finding in this section that shows just how politically polarized environmental science has become. When Pew compared the percentage of people with low, medium, or high science knowledge who say their view of environmental researchers is mostly positive, they found that science knowledge didn’t matter for Republicans. Only 39-41% of Republicans did, regardless of level of science knowledge, while the percentage increased from 48% of Democrats with low science knowledge to 89% among Democrats with high knowledge. As for the percentage of respondents saying that environmental researchers provide fair and accurate informational all or most of the time, the results with the same. For Republicans, only 18-20% agreed with this statement, regardless of their level of science knowledge. Among Democrats the results increased from 29% (low science knowledge) to 65% (high science knowledge). In other words, among those who are or lean Republican, level of scientific knowledge doesn’t affect their opinion of environmental scientists, and similar results were found when the same questions were asked about environmental practitioners.
Truly, climate science denialism and denial of environmental science appear to have become inseparable from GOP ideology and identity, and that’s a huge problem given that it hasn’t always been that way. Remember, the EPA was created during the Nixon administration, and as recently as the 2004 election cycle and even beyond, many Republicans were not hostile to climate science and accepted that human activity is contributing greatly to climate change.
There also remains distrust of science on the Republican side in other areas, just not as strikingly different from that of Democrats. For instance, only 17% of Republicans compared to 29% of Democrats agree that nutrition research scientists give fair and accurate information most of the time; oddly enough, there’s essentially no difference in responses to the same question about dietitians (Republicans, 46%; Democrats 49%). There’s a similar pattern in terms of the percentage of respondents agreeing with this statement when applied to medical research scientists, with only 29% of Republicans agreeing, while 35% of Democrats agree. Partisan differences notwithstanding, I was appalled at how low both percentages were. When the same question was asked about medical doctors, 46% of Republicans compared to 52% of Democrats agreed.
So, although the overall view of scientists and physicians is favorable, and most people say they trust them, there are some big problems. When minorities don’t trust scientists, that’s a problem. When a whole political party rejects well-established scientific findings as part of its ideology, that’s a problem. When so few people trust certain branches of science, such as nutritional science, that’s a problem.
What to do about trust and mistrust of scientists and physicians
Overall, this survey provides a rather mixed picture of Americans’ trust in scientists and physicians. While overall trust is high, there are pockets of major mistrust based on scientific discipline, political differences, and race. Fortunately, the survey’s findings suggest some strategies that might alleviate that mistrust.
One finding is that people would be more likely to trust scientific findings if researchers were to make their data publicly available. Specifically, 57% would be more likely to trust research if the data were publicly available, with 35% saying it wouldn’t matter and, bizarrely, 8% saying it would make them trust the research less. Open access to data is, of course, more achievable. For instance, to public genomics data, most journals require that the raw data be deposited in one of several genomics databases, and promising to do so is often required more often as a condition of funding by grant agencies. That’s an outlier, though. Open access to data is still not required in the vast majority of scientific disciplines, although the proliferation of online supplemental data sections for journal publications does make some progress in addressing the open access question. Of course, as a scientist myself, I have to wonder how far it is practical to go with this strategy. Does anyone want to see my experiment where I forgot to add one reagent and therefore got a bizarre result?
Similarly, 52% would trust research findings more if the research were reviewed by an independent committee, with 37% saying it would make no difference, and 10% saying it would make them trust the research less. Of course, I can’t help but wonder what the heck an “independent committee” would be. Pew doesn’t define it, and those who know enough about science know that peer review is essentially a small independent committee of two to four reviewers (most of the time), while study sections that review grant applications tend to be large committees of a dozen or more scientists. As they say, the devil is in the details, and it’s not clear just what form of an “independent committee” to review research would increase trust in the results of that research.
Funding sources matter, too:
Industry funding stands out as a factor Americans say leads to lower trust. A majority of Americans (58%) say they trust scientific findings less if they know the research was funded by industry groups.
The effect of government-funded research is less clear. About half of U.S. adults (48%) say learning that a study has been funded by the federal government has no impact on whether they trust its findings. The remainder is closely divided between those who say government funding decreases their trust (28%) and those who say it increases their trust (23%).
Similar factors inspire public trust in practitioners. About two-thirds of the public (68%) say they are more likely to trust practitioners’ recommendations more if that practitioner is open to getting a second opinion. About one-quarter (23%) say a practitioner’s willingness to get a second opinion makes no difference, and just 7% say it decreases their trust.
Among practitioners, the answer is obvious: We really have to be a lot more transparent about our industry ties and acknowledge that even small gifts can influence our thinking. I’ve heard more doctors than I can remember vehemently deny that a dinner or gift from a drug rep influences them, but those denials are not consistent with what we know from science about human psychology.
Unfortunately, more government funding of research, while in my opinion a desirable thing, is not desirable to all:
Opinions about government-funded research differ by politics. Among conservative Republicans, just 9% say that government funding increases their trust in research findings, while 41% say it decreases their trust. In contrast, liberal Democrats are more inclined to say government funding increases (34%) rather than decreases (21%) their trust in scientific research.
These findings are in keeping with political divides over support for federal spending on scientific research and an array of other government policy and spending priorities.
As long as there remains a partisan divide on the question of research funding, I don’t know that scientists can do anything other than try to be as transparent as possible about their funding sources, because increasing industry funding risks increasing mistrust among liberals and increasing government funding risks increasing mistrust among conservatives.
In the end, we as scientists and physicians should be heartened that we are among the most trusted groups in the US. We should also be concerned and perhaps even alarmed at the warning signals contained in this survey sounding the alarm about public distrust of specific scientific disciplines by specific subgroups of Americans. More transparency of funding sources and industry ties could help alleviate some of that distrust, as could more open access of data, but the specifics of how we achieve these things will not be simple or uncontroversial.

Sensible Medicine: Medical Misinformation and Medical Groupthink From the Medical Establishment
We all have biases, including in-group loyalty. It’s often easy to see such bias in others, though nearly impossible to see in oneself. As such, it’s both normal and dangerous to imagine that only those who disagree with you are vulnerable to groupthink, while you are perfect beacon of independent, rational thought- along with everyone who agrees with you.
Jonathan Howard on August 11, 2023
Link

Sensible Medicine is a monetized Subtstack that spreads overt anti-vaccine misinformation. Though the doctors who run it claim to be “medical conservatives”, they mostly took a medically radical approach to a new, mutating virus, treating it as an overhyped, predictable entity from the pandemic’s earliest days. As the pandemic progressed, they repeatedly declared it over and encouraged the mass infection of unvaccinated youth by spreading blatantly bogus statistics and treating abnormal lab values after vaccination as a fate worse than death from COVID. A recent article there by Dr. Marty Makary gives us an opportunity to delve into their more pernicious misinformation techniques.
Dr. Makary’s article was ironically titled Does Anyone Still Care About Data? Though it was about new treatments for Alzheimer’s disease, it contained the following paragraph:
America’s Lost Art—the Critical Appraisal of Research
We witnessed similar medical groupthink during Covid. Throughout the pandemic, the medical establishment never talked about Covid’s leading modifiable risk factor—obesity. Instead, they had a singular, albeit important, pharmaceutical solution. They indiscriminately told everyone to get multiple vaccine doses, regardless of whether a person already had high levels of Covid antibodies from natural immunity, or if a healthy young male already had three vaccine doses. During Covid we saw naked allegiance to political figures, regardless of the data. Doctors and the public alike were quick to cite studies to support their ideas, even if the methodological quality would not meet entry criteria for a 7th grade science fair contest. Conversely, research that did not support a forgone conclusion was downplayed, ignored, or censored.
It may not seem that way, but there’s a lot to unpack there. It’s full of misinformation, and every accusation is a confession. Its aim is to create mistrust and resentment, not to inform and educate. Get comfortable and we’ll examine it one piece at a time. I apologize for the fact that:
The amount of energy needed to refute bullshit is an order of magnitude bigger than that needed to produce it.
“We witnessed similar medical groupthink during Covid.”
First note that Dr. Makary wrote in the past tense, about events “during Covid”, as if the virus is gone. Of course, this is false. While hospitals and morgues haven’t been overwhelmed in a long time, SARS-CoV-2 is infecting, injuring, and killing people every day. Dr. Makary, who still discusses the pandemic constantly, has been predicting the end of the pandemic since at least December 2020 when he said, “we may only need to get an additional 20% of the population immunized by February or March to really hit those 70% herd immunity levels.” He’s never wavered from this position. In February 2021 he penned as essay titled “We’ll Have Herd Immunity by April,” and another the next month titled “Herd Immunity Is Near, Despite Fauci’s Denial“. In May 2021, he said he was right, claiming that “most of the country is at herd immunity”. He wrote articles in the NY Post that month titled, “Risk of COVID is Now Very Low — It’s Time to Stop Living in Fear: Doctor” and “Don’t Buy the Fearmongering: The COVID-19 Threat is Waning“. Dr. Makary later referred to the Omicron variant as “omi-cold” and “nature’s vaccine.” Anyone who cared about data knew that Dr. Makary was wrong about all this.
Does anyone still care about data?
Dr. Makary also sets the stage by signaling that he’s a brave, maverick doctor, while his critics are sheep, incapable of independent thought, just “groupthink”. There’s a grain of truth here. Though we at SBM are not afraid to disagree with each other in public, we all have biases, including in-group loyalty. It’s often trivial to see such bias in others, though nearly impossible to see in oneself. As such, it’s both easy and dangerous to imagine that only those who disagree with you are afflicted by groupthink, while you are perfect beacon of independent, rational thought- along with everyone who agrees with you.
Though the doctors behind Sensible Medicine portray themselves as immune to “groupthink”, they largely marched in lockstep during the pandemic, objecting to most measures to control COVID and minimize its dangers. To pick one example amongst many, they were united in their belief that the pandemic was ending in early 2021, and that ridicule was an appropriate response to anyone who disagreed. Their sarcastic mockery occurred even during the pandemic’s deadliest month, January 2021, while over 3,000 Americans were dying of COVID daily, and even on what turned out to be the literal peak of the deadliest wave, January 16, 2021. Over 700,000 Americans have since died of COVID.
Medical groupthink during COVID
When herd immunity failed to arrive, Sensible Medicine doctors only further encouraged groupthink by repeatedly shaming and pathologizing the small number of nonconformists who take measures to avoid repeated infections. When the Omicron variant emerged, Dr. Makary was fearful of a “wave of fear”, saying it could fuel a “pandemic of lunacy”. Dr. Prasad, who started the pandemic saying personal attacks were “worse than COVID”, similarly weaponized mental illness to stigmatize cautious people. In June 2023 he said:
Medhi Hasan is an idiot. Of course we have to learn to live with covid. Everyone is living with it. That is precisely what happened. No one gives a shit about COVID anymore & no one sane is taking any precautions. It even bores me & I know fully how wrong the establishment was.
Talk about groupthink.
Dr. Prasad, who also still discusses the pandemic constantly, even formalized his dedication to groupthink by asking his followers to publicly pledge to refuse more COVID vaccines. A request for a written, public display of conformity is a classic influencer technique, described by Robert Cialdini who said:
Commitments are most effective in changing a person’s self-image and future behavior when they are active, public, and effortful.
Medical groupthink during COVID
“Throughout the pandemic, the medical establishment never talked about Covid’s leading modifiable risk factor—obesity.”
Next, Dr. Makary signals that he’s an outsider, and therefore not responsible for our pandemic disaster in any way. Parroting Dr. Prasad, who said “I know fully how wrong the establishment was”, Dr. Makary also wants his audience to believe he’s not part of the “establishment”. I’m with you, not them.
Talk about groupthink.
Of course, this is false. Dr. Makary is the Mark Ravitch Chair in Gastrointestinal Surgery at Johns Hopkins School of Medicine, and teaches public health policy as Professor of Surgery and Public Health at the Johns Hopkins Bloomberg School of Public Health. In 2018, he was elected to the National Academy of Medicine. He was well-known before the pandemic, particularly for his claim, debunked by Dr. Prasad, that medical error is the third leading cause of death. He has 234,000 Twitter followers, and was a ubiquitous media presence this pandemic. He was on Fox News hundreds of times, declaring the end of the pandemic and spreading anti-vaccine misinformation. According to an article from Media Matters from December 2021 about these appearances:
Makary has been described as “a darling of the anti-vaxxers” for contradicting current vaccine consensus among medical professionals. Makary has also written that there is “no compelling case” for vaccinating “healthy” children and claimed that he is “not aware of a single healthy child in the U.S. who has died of COVID-19 to date” – discounting all cases that involved disabled children with underlying health conditions.
Despite Makary making numerous false predictions (saying, for example, that the United States would reach herd immunity by April) and being called out by other public experts for downplaying the risk of infectious disease, Fox regularly invited him back on-air in 2021 to undermine public health efforts:
In March on The Story with Martha MacCallum, Makary predicted there was low risk of new variants causing COVID-19 surges, and on Your World with Neil Cavuto, he accused public health experts of “cry[ing] wolf” over the threat of new variants.
In June on America Reports, Makary went so far as to complain that discussion of new variants is “fearmongering” to “manipulate people to get vaccinated,” saying, “I’m for vaccines, but this has turned into a tool to try to coax people into it.”
In June on Special Report, Makary claimed it’s time to “move on and live a normal life” comparing COVID-19 cases to influenza: “right now we’re about at 150th the daily cases of a regular seasonal flu in the middle of that flu season. So people have a distorted perception of risk.” According to Makary’s own employer, Johns Hopkins, COVID-19 has a mortality rate 10 times higher that of the flu.
On Your World with Neil Cavuto in November, Makary undermined efforts to vaccinate children, saying kids with natural immunity should not be vaccinated. He went on to fearmonger about potential side effects of vaccinations.
On Fox News Primetime in November, Makary claimed to be pro-vaccine but said he stands against “obsolete” vaccine mandates, especially for children. Makary falsely asserted that “public health officials have brushed under the rug the fact that kids have died from the vaccine, it’s rare” citing myocarditis occurrences in young boys. (However the real risk of myocarditis is from COVID-19, not vaccines, and most people who develop myocarditis recover quickly.)
Dr. Markary was also the editor of Medpage Today and spread his misinformation message in the Wall Street Journal, New York Post, and Washington Post. He testified before Congress on more than one occasion and had private audiences with Republican congressional doctors. He was the COVID advisor to the Governor of Virginia. You can even hire him to speak for $20,000-$30,000.
He is definition of the medical establishment.
Medical groupthink from the medical establishment
Having falsely told his readers he’s not part of the “establishment”, Dr. Makary then claims they “never talked about Covid’s leading modifiable risk factor—obesity”. Of course, this is false. The CDC listed “Severe obesity (body mass index [BMI] of 40 or higher)” as a risk factor for severe COVID as early as April 2, 2020 and they later published studies on the subject got that widespread media attention. They said:
As clinicians develop care plans for COVID-19 patients, they should consider the risk for severe outcomes in patients with higher BMIs, especially for those with severe obesity.
They were not alone. As early as March 2020, Dr. Anthony Fauci said:
It’s so clear that the overwhelming weight of serious disease and mortality is on those who are elderly and those with a serious comorbidity: heart disease, chronic lung disease, diabetes, obesity, respiratory difficulties.
He elaborated in 2021, saying:
The other thing I believe, is that obese people, particularly morbidly obese people, have a much higher rate of the other underlying comorbidities, which get a serious outcome. That’s diabetes, hypertension, heart disease—the incidents in a morbidly obese person of those other diseases is much higher than the general population.
Similarly, Dr. Francis Collins, the former director of the NIH, said in June 2020:
We also know that chronic illnesses play a role in who’s going to have a severe case. Certainly hypertension and diabetes and obesity are in there as well.
In fact, a simple Google search returned around 100 articles by the “establishment” on obesity and COVID from throughout the pandemic. My university published an article on April 16, 2020 titled “The New York Times: Obesity Associated with Severe Coronavirus Disease, Especially in Young Adults“. One from Dr. Makary’s university on June 1, 2020 was titled “Obesity a Major Risk Factor for COVID-19 Hospitalizations.” Harvard, Yale, Stanford, UCSF, The University of Chicago, and dozens of other medical schools wrote about obesity and COVID. So did the American Medical Association, the American Academy of Pediatrics, the American Heart Association, the Endocrine Society, and numerous other medical organizations. Major newspapers discussed it. I also discussed obesity as a risk factor for severe disease in children, noting that nearly all who perished from the virus were robbed of decades of life. It’s hard to find examples of the “establishment” who completely ignored the relationship between obesity and COVID
“Throughout the pandemic, the medical establishment never talked about Covid’s leading modifiable risk factor—obesity.”
Maybe the “establishment” should have talked about obesity more- Dr. Makary has just seven Tweets on the topic, the first from December 2020, the last from May 2022 -but it wasn’t exactly a top classified secret that it posed a risk for severe COVID. Nonetheless, Dr. Makary imagines that a large number of obese people were totally unaware of this risk, and that many of them could have have lost weight, if only the “establishment” had been more vocal on the topic. One overweight friend reacted to Dr. Makary by telling me:
Do people really think fat people don’t know they are fat and would instantly be thin just because some doctors says, “Oh, you should lose weight”?
People do actually think that, and it’s absurd. Many of my patients are overweight. They are not ignorant of obesity’s health risks, and nearly all tried everything for years to lose weight. While weight-loss is crucial and possible for many people, it’s not easy or instant for anyone, and a bit more encouragement from the “establishment” was not the secret sauce anyone was missing. Protecting obese people, who could not lose weight instantly three years ago is one reason we went to such lengths to control the the virus. 42% of Americans are obese, their lives are not expendable, and the “establishment” could not have reversed obesity on a dime in March 2020. If Dr. Makary disagrees, then he should explain why he failed to cure obesity in Virginia during his stint as that state’s COVID advisor.
And let’s pause and look what Dr. Makary has communicated in just two short sentences. He’s signaled that COVID is over, denigrated his critics as the “establishment”, and claimed they were incapable of independent thought. He absurdly implied that obese people were ignorant of their condition, and that “Covid’s leading modifiable risk factor” could have been drastically improved if not for the apathy of the “establishment”. Moreover, he disrespected his readers by spreading easily-refutable misinformation, claiming that “Throughout the pandemic, the medical establishment never talked about Covid’s leading modifiable risk factor—obesity.” Throughout the pandemic! Never! Either Dr. Makary didn’t bother with basic fact checking or he didn’t care that he was spreading a blatant falsity. However, his goal wasn’t to communicate accurate information. Like most Sensible Medicine writers, his aim was to trigger anger and groupthink.
Throughout the pandemic! Never!
Of course, legitimate criticisms of the “establishment” are necessary and vital, and we here at SBM don’t shy away from criticizing medical boards, regulatory agencies, academic fraud, problematic drug approvals, as well as many individual “leaders” of American medicine. Some of the best work in this area comes from those truly outside the medical establishment, such as the Death Panel Podcast. However, while mistakes were made and some bad faith actors purposefully undermined all efforts to contain the virus, the vast majority of public health officials were people you’ve never heard of, working on the ground, doing the best they could to save lives in an unprecedented situation. Whatever their errors, it wasn’t easy to have real-world responsibility during the pandemic. As the bodies piled up, tough decisions had to be made in real-time with incomplete information. The right answers weren’t always obvious. This was everyone’s first pandemic and SARS-CoV-2 was a new virus, after all.
Moreover, public health agencies hadn’t been showered with cash and resources prior to the pandemic. Quite the opposite, actually. Public health officials couldn’t have done everything that’s now being demanded of them. When morgues were overflowing, they didn’t have the resources to launch a nationwide obesity awareness campaign.
However, asinine claims that the “establishment” could have tackled obesity in March 2020 have a broader purpose. They are part of a larger movement to rewrite history and minimize the enormous challenges the “establishment” faced at the pandemic’s start. Of course, this is just another way to minimize COVID itself. Some influential people want everyone to forget that refrigerated trucks were needed to store dead bodies outside my hospital. Doctors who greatly underestimated the virus at the start and who never treated COVID patients themselves, now seek to absolve their calamitous errors by blaming the “establishment” for not fixing every societal ill in March 2020. It’s no big deal that I said we’d have herd immunity by April. People just needed to stop being fat, and the establishment is to blame for that, not me. Unsurprisingly, the original COVID minimizer Dr. John Ioannidis, pioneered this deflection technique in 2021, creating a list of what “really caused” millions of COVID deaths. In 2023, Dr. Makary parroted his exact language, calling obesity a “modifiable risk factor” to similarly imply it “really caused” many COVID deaths.
Talk about groupthink.
Dr. John Ioannidis on what “really caused” COVID deaths
With the “establishment” and obese people there to take the blame, these doctors then seek to convince us that all measures to control the virus were an overreaction, a position many of them formulated in March 2020, when they thought fewer than 40,000 Americans would die. For example, demonstrating yet more groupthink by parroting the doctors behind the Great Barrington Declaration, Dr. Prasad said in March 2022:
Masks, distance, lockdown, test often don’t work, if they work, only delay the inevitable great harm.
Of course it’s unambiguously good that measures to control the virus delayed the inevitable and allowed countless millions of people to avoid it until after they were vaccinated and medical care had improved. The vaccines aren’t perfect, but a fully vaccinated person who contracts COVID today has much better odds of a good outcome than someone who contracted it in April 2020 when hospitals were deluged and we had no idea how to treat it. According to one modeling study:
From December 2020 through November 2022, we estimate that the COVID-19 vaccination program in the U.S. prevented more than 18.5 million additional hospitalizations and 3.2 million additional deaths.
An obesity awareness campaign wouldn’t have had this impact.
In lieu of legitimate criticisms of the “establishment”, Sensible Medicine makes blatantly false statements about them and makes very difficult, even impossible tasks sound trivial. They want their readers to believe the “establishment” could have done nearly anything, but they were too stupid or apathetic to do many easy and obvious things. This too has a broader purpose. Their main goal is to get people to stop trusting the public health “establishment”, and you don’t have to take my word for it.
Unfortunately, Sensible Medicine is very successful in getting their audience to stop trusting the public health “establishment.” One commentator on Dr. Makary’s article said:
So sad. Medical community has gone crazy – and evil – since covid pandemic emerged.
This unbridled rage is par for the course for the grossly misinformed readers of Sensible Medicine, and that really matters. I encourage you to read the comments on an article titled “A Public Health Worker Looks Back at the Pandemic and How We Reacted to It,” by Matthew Brignall, a public health worker who was brave enough to work on the ground during during the pandemic and even braver to share his experience on Sensible Medicine. He wrote:
State and local health policies suppressed the spread of COVID-19 until we could develop an effective vaccine and get it into the arms of the most vulnerable people.
His reward for making sensible statements on Sensible Medicine was hundreds of comments calling him a “biofascist scum” and someone who helped “usher in tyranny around the globe.” Dr. Adam Cifu, a founder of Sensible Medicine, knew his readership and tried to prepare him for the backlash, but even Mr. Brignall was taken aback by the fury and groupthink, saying:
To be fair, Adam warned me, and I chose not to turn off the comments. But yeah, that shit was a cesspool. I think it is instructive to see how quickly the internet goes to lynch mob when the opinion is challenging.
Indeed, Sensible Medicine excels at creating a cesspool of resentment. I previously discussed an article there that lambasted the “establishment” for not doing an absurd, utterly impossible randomized-controlled trial (RCT) of hospital visitation policies at the pandemic’s start. Predictably, their readers, consumed by groupthink, responded not by agreeing to volunteer for future RCTs, but by sharing their fury and retribution fantasies against the “establishment”. “It was a purposeful dictate designed to wreck humanity,” said one. “These people need to be held accountable”, said another. “It was criminal in my mind”, said another. Another said:
To the MD that wouldn’t let her go “home” on her birthday, may he be forever riddled with guilt. May he suffer the loss of a loved one in the same manner. May he die alone.
Charming stuff. Meanwhile, Dr. Cifu, who was very concerned about the “tone” of those who corrected the factual errors of his Sensible Medicine colleagues, also appeared in the comments and said to its author, “Wonderful. Thank you so much Ben“.
All of this anger and mistrust had consequences in the real-world, and none of it was “wonderful”. RCTs require volunteers who trust clinical trialists, AKA the “establishment”. I was in a vaccine RCT and because I actually want them to be successful, I always spoke positively about them. In my first article this pandemic, I discussed my experience and said:
Having argued for years that vaccines are properly tested, I jumped at the chance to participate in a vaccine trial myself. I’ve always believed that people who participate in medical research are doing a noble thing. If no one volunteered for such research, medicine would never advance.
In contrast, I don’t recall a Sensible Medicine doctor ever promoting an actual RCT or encouraging their audience to enroll in one. However, advancing actual RCTs is not their goal, which is a shame. Some very simple RCTs couldn’t even get off the ground due to low enrollment. It turns out that actually doing an RCT is a lot harder than writing “do an RCT”, and podcasting doctors who excoriated the “establishment” for not doing RCTs of everything, made it harder for anyone to do an RCT of anything.
Beyond this, many public health officials, AKA the “establishment”, were threatened, harassed, and attacked during the pandemic, leading many to quit. One public healthcare worker sent me the following message:
To shield themselves from similar vitriol, Sensible Medicine doctors falsely portray themselves as outsiders, wryly observing the “groupthink” of the “establishment” from a distance. Though they absolutely are part of the “establishment”, they don’t consider it their job to solve obesity, run an RCT, or seemingly do much of anything, besides commenting from the sidelines, blaming others for not doing everything they imagined. “If only they’d listened to me, things would have been just fine,” they say in articles that should be titled “What I Would Have Done“.
Of course, none of us have to wonder what we would have done this pandemic. What you actually did is exactly what you would have done.
“Instead, they had a singular, albeit important, pharmaceutical solution.”
Having poisoned the well against the “establishment”, Dr. Makary then pivots to his standard anti-vaccine blather. According to Dr. Makary, the brainless drones who chose not to cure obesity, had a “singular” focus on vaccines. This is partially true. Dr. Fauci, an avid jogger himself, encouraged exercise in the summer of 2020. He also took a medically conservative approach to the vaccines, saying in February 2021:
But even if you’re one of the lucky few to have gotten a Covid-19 shot, it will be important to continue wearing a face covering until researchers can determine whether the vaccines prevent people from spreading the virus to others who aren’t vaccinated.
However, for over a year, the “establishment” has abandoned all measures to control the virus, aside from their singular focus on vaccines. Put another way, the “establishment” adopted essentially all the positions of Sensible Medicine doctors, who are, of course, the “establishment”. Indeed, it was the doctors at Sensible Medicine who had a “singular” focus on vaccines, claiming they obviated other measures to contain the virus almost immediately after they came out. Dr. Markary said in March 2021:
If anyone hears of a person hospitalized or dead from Covid-19 after full vaccination, please let me know. The data show that vaccines confer near perfect protection against death and hospitalization from Covid. Can’t we be honest about that?
Dr. John Mandrola said in March 2021, “The pandemic is essentially done….The vaccine squelches severe cases.” Later that year he said, “Have the vaccines not transformed SARSCOV2 into just another regular respiratory virus?” Though he later said such comments were the “worst misinformation” Dr. Prasad said in January 2021:
You will be less infectious, dramatically less infectious, less of a threat to others after vaccination. Of course, no surprise, that’s why we’re vaccinating, right?
In February 2021, just after the deadliest month of the pandemic, Dr. Prasad mocked the New York Times for suggesting that vaccines alone might not end the pandemic. “It was just completely off, and just another squeeze of fear,” he said. That same month he said, “we know the vaccine is 100% effective against protecting against bad outcomes for grandparent,” and in May that year, he claimed:
For somebody who’s already been fully vaccinated, they can wear the mask out of solidarity or in a symbolic sense, but their wearing a mask indoors is not benefiting anyone else. There’s an infinitesimally low probability of even having an infection that can be detected on a PCR test, let alone being able to spread it to someone.
This was pure groupthink from establishment doctors who rejected medical conservatism and now blame the “establishment” for their words. Thankfully, I avoided this groupthink and rejected a singular focus on vaccines. In April 2021, I said,
Im very very optimistic vaccines will cut transmission of disease. I’m not certain of it. This is why I wear a mask still in public and when treating covid patients.
A singular focus on vaccines
Notice too how Dr. Makary calls vaccines a “pharmaceutical solution”. While this obviously isn’t wrong, this framing was a conscious choice to spread fear and doubt. If you think I’m overstating the case, consider the subtitle of an essay of his that spread misinformation about the COVID vaccine in pregnancy:
The Agency (CDC) did exactly what big Pharma wanted, using groupthink instead of insisting on data for their recommendations.
That sentence could appear on anti-antivaccine site about any vaccine. It’s standard anti-vaxx balderdash. It’s no secret that many Americans distrust the pharmaceutical industry, often with good reason. However, by telling people vaccines = pharma, Dr. Makary sought to reinforce this negative association for his readers. It worked. One of his commentators said:
Pharma won’t allow lifestyle change to trump sales, so we can’t depend on them. Many health organizations make $ only when there are patients, so we won’t see preventatives from whole organizations. Thank god for individuals who ignore the business model and want to help individual patients.
Dr. Makary could have said vaccines are an “immunological solution” and presented the copious evidence they reduced grave outcomes. But having greatly oversold them in 2021, Sensible Medicine doctors are loathe to frame them in a positive light today. Of course, by decreasing hospitalizations, COVID vaccines saved the healthcare system enormous sums of money, including a large amount spent on pharmaceutical products. Those of us are actually opposed to mere “pharmaceutical solutions” to COVID, embraced vaccines while being honest about their flaws. At least Dr. Makary conceded vaccines are “important”.
“They indiscriminately told everyone to get multiple vaccine doses, regardless of whether a person already had high levels of Covid antibodies from natural immunity, or if a healthy young male already had three vaccine doses.”
Dr. Makary then claims “they indiscriminately told everyone to get multiple vaccine doses”. Of course, this is false. You may feel the “establishment” should have left more unvaccinated people vulnerable to the virus, but they didn’t come to their policies on a whim. The Advisory Committee on Immunization Practices (ACIP) had multiple lengthy, public meetings to review the data and discuss every vaccine dose. In fact, the “establishment” disagreed on boosters initially. In September 2021, CDC director Dr. Rochelle Walensky overruled the ACIP, which had voted 9-6 against suggesting boosters to health care workers and other frontline workers. Several vaccine regulators voiced their opposition in the Washington Post and ultimately resigned in protest, something Dr. Makary celebrated on social media.
So much for groupthink.
Moreover, many people felt the “establishment” dragged its feet in authorizing pediatric vaccines. There was a real price to paid for “precaution”. Of course, unless the “establishment” ran a massive RCT for every tweek to the vaccine, Sensible Medicine doctors would always accuse them of acting “indiscriminately”, as if only the vaccine posed a threat, never the virus.
The belief that the virus was essentially harmless for all but the most vulnerable people may explain why Sensible Medicine doctors indiscriminately told everyone to contract it multiple times. Even though no RCT shows this is safe, Dr. Prasad said in 2023, “Repeat infections are inevitable. More the longer you live. Nothing can be done about it.” Dr. Cifu expressed the exact same sentiment in May 2022, saying:
We now need to accept that this is here to get infected with again and again.
Talk about groupthink.
Notice as well that Dr. Makary views “antibodies from natural immunity” as a valid reason to forego vaccination. Sounding like a pre-pandemic anti-vaxxer gushing over measles, Dr. Makary acted like cheerleader for SARS-CoV-2 much of the pandemic – “natural immunity wins again“- claiming that vaccines and “natural immunity” would usher in herd immunity by April 2021. Even after this prediction failed miserably and immune-evading variants arrived, he continued to lament that “policymakers” did not glorify “natural immunity”. “It’s okay to have an incorrect scientific hypothesis. But when new data proves it wrong, you have to adapt”, he wrote, scolding people who didn’t claim herd immunity arrived in May 2021.
Of course, the power of “natural immunity” was that it killed over a million Americans. “Natural immunity” injured millions more, and isn’t as powerful as hoped for these survivors. Moreover, there is evidence that vaccination after infection was beneficial at limiting hospital admissions and severe disease. Sensible Medicine readers won’t know this.
Beyond this, Dr. Makary implies there’s something nefarious about “multiple vaccine doses”, though he doesn’t say what that is. Of course, most vaccines require multiple doses. Children receive four polio vaccinations by the time they are six. They receive five doses of the DTaP vaccine by this age. The flu vaccine is given annually. Does Dr. Makary blame the “establishment” for this?
Though it would have been great, there was never any guarantee that the first two COVID doses would provide lifelong protection against a new, mutating virus. However, even if the “establishment” ended up suggesting a vaccine dose to someone who didn’t need it, they had to make recommendations with less-than-perfect data in the middle of a rapidly changing pandemic. That’s medicine. It’s a fantasy to imagine that we will always have a tidy RCT to resolve every question. The risks will not always be perfectly known and letting children, especially unvaccinated ones, repeatedly get COVID is an example of an unknown risk.
Nonetheless, it’s true that some people got a vaccine dose they probably didn’t need. However, Dr. Makary didn’t explain why, in a pandemic where 1.1 million Americans died and millions more were injured, this was such a cause for concern. Why was it really so disastrous if a healthy young male chose to get a 4th dose? I’ve discussed the risks of the vaccine many times previously, and while tragic, catastrophic reactions, including deaths, have occurred, they are literally less than one in a million events. In contrast, a study from May 2022, estimated that nearly 319,000 deaths could have been prevented if all American adults had gotten vaccinated. Dr. Makary did not bemoan the fate of the people who suffered needlessly because they refused a vaccine dose they did need- including the first booster and bivalent booster– and the deceased are not here to extol the power of “natural immunity”.
The power of “natural immunity”
“During Covid we saw naked allegiance to political figures, regardless of the data.”
Dr. Makary was not wrong about this, nor was he the only Sensible Medicine doctor to portray himself as floating above the political fray. Dr. Prasad and Dr. Jeffrey Flier shared a similar concern at the start of the pandemic in an essay titled “Scientists Who Express Different Views on Covid-19 Should be Heard, not Demonized“. This authors said they were “concerned with personal attacks and general disparaging comments” and concluded.
Society faces a risk even more toxic and deadly than Covid-19: that the conduct of science becomes indistinguishable from politics.
In an article titled “Politics and Medicine is a Bad Idea” Dr. Mandrola similarly warned of the dangers of mixing politics and medicine. He said, “our medical training provides us no expertise on policy”. “There are many downsides when healthcare professionals overstep our expertise.” he added.
Talk about groupthink.
Dr. Mandrola ended his essay by saying:
We should remain like a neutral judge. Always. The people can decide the best policy.
Despite this clear, unambiguous conclusion, Dr. Mandrola did not mention his Sensible Medicine colleague Dr. Prasad. This is not for lack of opportunities. Dr. Prasad, an adult oncologist whose medical training provided him no expertise on policy, repeatedly compared school closures to the Iraq War and wrote over 50 pugilistic Tweets on the evils of teachers unions. When schools in Philadelphia closed a Friday due to poor air quality from forest fires, he said:
I really hope that in return for your repeated failure to do what is right for children we move to a voucher system that crushes your failed district and destroys your union system that always places adult interests over children.
The schools reopened Monday.
Dr. Prasad also frequently villainized “Democratic cities”, “left wing politics” “NYTimes reporters” “liberal cities” and “Fauci” . One of many emotive Tweets attacking “dems” said:
Dems have become the corporatist party. Pfizer basically writes EUA orders based on mice data. At the worst moment of the pandemic Dems abandoned kids, esp poor minority kids to side with teachers unions. Instead of hearing criticism, they preferred to censor Twitter & YouTube.
Though I won’t say that his behavior was “more toxic and deadly than Covid-19”, Dr. Prasad also revelled in demonizing scientists who expressed different views. He said that RFK Jr, a presidential candidate and anti-vaccine crank with a body count, would “destroy”, “flatten” and “obliterate” Dr. Peter Hotez, a highly-respected vaccine scientist who received anti-semitic death threats and was harassed at home for wisely refusing to perform a verbal joust with RFK Jr. on a podcast.
Talk about the conduct of science becoming indistinguishable from politics.
The conduct of science becomes indistinguishable from politics.
RFK Jr. was not the only anti-vaccine politician to receive praise from Dr. Prasad. In May 2023 he said that:
DeSantis Florida had the best COVID-19 policy in the United States. Balancing competing risks and freedom.
The data argues otherwise. A recent article titled The Steep Cost of Ron DeSantis’s Vaccine Turnabout shows how anti-vaccine doctors with naked allegiance to political figures ignored the data, costing thousands of young Floridians their lives. It said:
At the same time, though, the governor was embracing more extreme views, including those of Dr. Scott W. Atlas, a Stanford neuroradiologist with no expertise in infectious diseases. Dr. Atlas was a frequent commentator on Fox News when Mr. Trump named him to his Covid task force in August 2020.
Both he and Dr. Bhattacharya argued that people who were not at risk of severe consequences should not face Covid restrictions. If they were infected, they would develop natural immunity, which would eventually build up in the population and cause the virus to fade away, they said….
Dr. Atlas, however, argued that the virus was not dangerous to an overwhelming majority of Americans. Both he and Dr. Bhattacharya said the Covid death rate for everyone under 70 was very low. Dr. Atlas claimed that children had “virtually zero” risk of death. Neither man responded to requests for comment…
While Florida was an early leader in the share of over-65 residents who were vaccinated, it had fallen to the middle of the pack by the end of July 2021. When it came to younger residents, Florida lagged behind the national average in every age group.
That left the state particularly vulnerable when the Delta variant hit that month. Floridians died at a higher rate, adjusted for age, than residents of almost any other state during the Delta wave, according to the Times analysis. With less than 7 percent of the nation’s population, Florida accounted for 14 percent of deaths between the start of July and the end of October.
Of the 23,000 Floridians who died, 9,000 were younger than 65. Despite the governor’s insistence at the time that “our entire vulnerable population has basically been vaccinated,” a vast majority of the 23,000 were either unvaccinated or had not yet completed the two-dose regimen.
The two images below, just six weeks apart, summarize the largely preventable tragedy well.
DeSantis Florida had the best COVID-19 policy in the United States
The unchecked spread of the virus also meant many schools couldn’t stay open in Florida. “It’s just not sustainable when you don’t have the people to run the school,” said one principal. Another superintendent said during the Delta wave:
COVID numbers are currently on the climb and having an impact on students, teachers, and on the support staff on whom we rely so heavily – particularly our food service workers and bus drivers.
This is the state and governor that Dr. Prasad said had “best COVID-19 policy in the United States”.
Dr. Prasad even defended the Surgeon General of Florida, Dr. Joseph Ladapo, saying his critics of his anti-vaccine data fraud were “stupid.” “Ladapo is closer to correct than his critics”, he said about a doctor who altered data to make the vaccine appear more dangerous than the virus.
Talk about naked allegiance to political figures, regardless of the data.
Naked allegiance to political figures regardless of the data
“Doctors and the public alike were quick to cite studies to support their ideas, even if the methodological quality would not meet entry criteria for a 7th grade science fair contest.”
Dr. Makary then laments that “Doctors and the public alike were quick to cite studies to support their ideas, even if the methodological quality would not meet entry criteria for a 7th grade science fair contest.” Again, he’s not wrong. Many bad studies were published this pandemic, and Sensible Medicine doctors were eager to amplify those that supported their ideas. For example, Drs. Prasad and Makary both promoted a horrible, retracted anti-mask study written by anti-vaccine activists, and Dr. Makary’s enthusiastic Tweet about it remains uncorrected to this day. He said:
Kids need fresh air! A new JAMA study describes consequences of elevated carbon dioxide levels in kids and hypercapnia from inhaled air from the dead space a mask creates.
Dr. Makary did more than amplify bad studies that supported his ideas, he authored one himself. In an editorial tilted “Think Twice Before Giving the COVID Vax to Healthy Kids“, Dr, Makary said:
In reviewing the medical literature and news reports, and in talking to pediatricians across the country, I am not aware of a single healthy child in the U.S. who has died of COVID-19 to date. My research team at Johns Hopkins partnered with FAIR health to study pediatric COVID-19 deaths using approximately half of the nation’s health insurance data. We found that 100% of pediatric COVID-19 deaths were in children with a pre-existing condition, solidifying the case to vaccinate any child with a comorbidity.
Dr. Makary’s report was titled “Risk Factors for COVID-19 Mortality among Privately Insured Patients: A Claims Data Analysis”. It was published as a “white paper,” not in a medical journal. Though he claimed it was a study of “pediatric COVID-19 deaths”, the words “child,” “children,” and “pediatrics” do not appear in it, though there is a discussion of Alzheimer’s disease in relation to COVID. The paper reported on 2,753 people who died of COVID from April 1, 2020, through August 31, 2020. Of these, 0.11% deaths occurred in children under 18-years. This means Dr. Makary’s supposed study of “pediatric COVID-19 deaths” reported on just three deceased children. It presented no data about them beyond noting they were under age 18 years and had some underlying condition.
Would a 7th-grader at a science fair use a sample size of three to claim that zero healthy children had died of COVID in the entire country? If they did, they’d be wrong. Sadly, healthy children had died of COVID when Dr. Makary made his false claim, something a 7th grader could have easily discovered with a simple Google search.
Of course, the lives of vulnerable children are not less valuable than healthy ones.
“Conversely, research that did not support a forgone conclusion was downplayed, ignored, or censored”.
Dr. Makary was not wrong to say that “research that did not support a forgone conclusion was downplayed, ignored”. I’ve written previously (here, here, and here) on how Sensible Medicine writers cherry-pick information to minimize COVID’s impact on children, as well as to fear monger about the vaccine and hide its benefits. I previously said
If ten data points are needed to understand the risks and benefits of the vaccine, they cherry pick the three that support their position.
Indeed, most Sensible Medicine doctors started with the conclusion that unvaccinated children should contract COVID, and worked backwards to selectively present the “evidence” only that supported their mass infection. To my knowledge, Dr. Makary has never shared any of the dozens of studies showing the vaccine has helped limit rare, but grave harms in children. Sensible Medicine doctors pretend these studies do not exist.
Dr. Makary ends with conspiracies, though he does not say which research was “censored”. In fact, his paragraph didn’t contain a single link to support any of his claims. However, in March 2022, he complained that the “establishment” rejected his findings on obesity and COVID. He said:
Unfortunately a lot of medical journals were heavily curating the establishment narrative. Our Hopkins study showing obesity was a major risk factor for Covid mortality was ready for publication in July 2020 but was rejected at 7 medical journals over a 1.5 year period.
What does Dr. Makary think the “establishment” has to gain by covering up a link between obesity and severe COVID. Why does he think the editors of multiple medical journals agree to suppress such an important finding?
If indeed the “establishment” did seek to “curate” medical journals to hide this link, they did a poor job of it. In addition to the articles I linked to previously, on April 1, 2020 the editors of the journal Obesity wrote an editorial titled “COVID 19 and the Patient with Obesity – The Editors Speak Out” in which they said:
Persons with obesity around the world are already at high risk for severe complications of COVID-19, by virtue of the increased risk of the chronic diseases that obesity drives.
The New York Times wrote an article in April 2020 titled “Obesity Linked to Severe Coronavirus Disease, Especially for Younger Patients” that referenced research at NYU. It said:
One of the largest U.S. studies to identify obesity as a prominent risk factor analyzed data from more than 4,000 Covid-19 patients who sought care at NYU Langone Health between March 1 and April 2.
It also discussed:
Another NYU Langone study, which focused on patients under the age of 60, found that those with obesity were twice as likely to be hospitalized and were at even higher risk of requiring critical care.
In fact, by July 2020, enough data had accumulated that to allow for an review on the topic titled “The Potential Impacts of Obesity on COVID-19“. By this early date, the author was able to reference several papers and conclude:
Obesity has been found to be a strong independent risk factor for hospitalisation in COVID-19, and to cause or exacerbate a variety of comorbidities which have been proven to be associated with increased morbidity and mortality among COVID-19 patients.
In September 2020, Science published an article titled “Why COVID-19 is More Deadly in People with Obesity—Even if They’re Young“. It quoted a doctor who said:
These were otherwise healthy, hard-working people. Their major risk factor for getting this sick was obesity.
Maybe seven medical journals rejected Dr. Makary’s paper for reasons that had nothing to do with censoring the link between obesity and COVID.
Let’s return to the title of Dr. Makary’s article, Does Anyone Still Care About Data? Many people do, and Sensible Medicine doctors taunt them by robotically levelling the same juvenile accusation of “groupthink”. This, of course, is just medical groupthink from the medical establishment.
And if the doctors behind Sensible Medicine want to prove me wrong, they have my blessing to publish this essay on their site.
Medical groupthink from the medical establishment.

When an antivax physician “dies suddenly”: The case of Dr. Rashid Buttar
Last month, Dr. Rashid Buttar, a prominent antivax “integrative medicine” practitioner, died suddenly. Because he hadn’t been vaccinated, antivaxxers struggled mightily to reconcile his death with their conspiracy theory about COVID-19 vaccines killing thousands “suddenly.” It turns out, however, that that Dr. Buttar had not been a well man since 2016 and was as much a victim of quackery as his patients had been.
David Gorsk on June 19, 2023
Link

A month ago, I was surprised to learn that Dr. Rashid Buttar had passed away, having seen chatter about it on social media that was ultimately confirmed by Sayer Ji on Twitter:
Starting at the one hour mark, Dr Buttar shared what lead to his steep health decline in Jan of this year: https://t.co/wU7YIOt2A9
— Sayer Ji (@sayerjigmi) May 20, 2023
As many of you might remember, Sayer Ji has been featured on this blog before. He was husband to Dr. Kelly Brogan, the anti-psychiatry, antivax physician and germ theory denier who rose to greater prominence after the COVID-19 pandemic hit, to the point where she was named one of the Disinformation Dozen. The two of them ultimately divorced, but both continued down their previous paths of spreading misinformation. In any event, I must admit to my shock over seeing that Sayer Ji announce the death of fellow “Disinformation Dozen” member Dr. Rashid Buttar.
As you will see (and can probably glean just from the Tweet and the linked video), Dr. Buttar’s story is a bit more complicated than the portrayal of a perfectly healthy fifty-something year old man having “died suddenly.”” Truth be told, I had had no idea that Dr. Buttar had been sick, and, as always, empathize with his family and friends, regardless of what he had done during his life. Also, I hadn’t realized that he was actually younger than I am, and it always bothers me to see people my age or younger die prematurely. As you will see, the story is about what antivaxxers claimed about Dr. Buttar’s health history after he had died compared to what his health history, as discoverable on social media and prior publications, actually was. Of course, it’s no shock that antivaxxers struggled mightily to blame his death on COVID-19 vaccination, even though they admitted that, as an antivaxxer, Dr. Buttar had never been vaccinated.
Any time one of their own dies suddenly or suffers a nonlethal health problem that they normally attribute to vaccines, antivaxxers have a problem: How do they explain it in the context of their conspiracy theory and misinformation about vaccines? Do they take a tragic event like the death of one of their unvaccinated antivax friends as potential disconfirming evidence, as a reason to question their narrative? I think you know the answer to that question. They find ways to double down, no matter how much they have to contort reality to do it.
So let’s compare the antivax narrative surrounding what happened to Dr. Buttar to what most likely happened. This discussion has been helped immensely by my discovery, thanks to Dr. Stephen Barrett, of an account found in an online book by “investigative medical journalist and esteemed health influencer Jonathan Otto entitled POISONED: A Deep Dive into the Mass Bioweapon & Envenomation Agenda, which promises, among other things, to tell you “the uncensored story of the late Dr. Rashid Buttar tragic story.””
Who was Dr. Rashid Buttar?
Before I get into the story of Dr. Buttar’s death, let me briefly recount the story of his life, specifically his longstanding role as one of the influential practitioners of “integrative medicine” quackery in the US. To be honest, I hadn’t paid much attention to Dr. Buttar in a while. The last time I mentioned him was last fall, when he was featured in a post about the ongoing assault by antivaxxers and promoters of unproven (and disproven) attacks on science-based medical regulation as an example of how promoters of quackery had been chipping away at science-based regulation of medical practice. Way back in 2007, Dr. Rashid Buttar made the news for calling the North Carolina Medical Board a “rabid dog” when it tried to discipline him for “unprofessional conduct” related to his cancer and antivaccine autism quackery. Let me just go back and quote what Dr. Buttar said in 2007, quoted by The Charlotte Observer and other newspapers:
Here was the sort of quackery that Dr. Buttar practiced up until his demise last month:
But Buttar believes most patients with cancer also have high levels of heavy metals, such as mercury, and other toxins, such as pesticides, because of environmental exposure. “Chelation works for the right things,” he said. “It’s not a panacea. But it works phenomenally well for chronic disease.” Buttar’s practice also has two hyperbaric oxygen chambers — and more on the way. The treatment, which floods the body with oxygen, is commonly used to accelerate wound healing. But Buttar uses it to fight cancer because, he says, cancer cells can’t thrive in an oxygen-rich environment. He acknowledges most of his treatments have not been proven by standard means, such as controlled clinical trials. But he said they have been effective in his practice over 10 years.
The story noted that Dr. Buttar’s therapy for cancer patients ranged from $40,000-60,000 for two months, and, of course, he didn’t take insurance. (Like Sayer Ji and his ex-wife, Buttar, too, was also one of the “Disinformation Dozen“.) As I noted at the time, unfortunately, Dr. Buttar’s practice and his war against the North Carolina Medical Board (which he basically won). Basically, he got a slap on the wrist, being temporarily restricted from treating cancer patients and children. The medical board did not do what was really warranted, strip him of his medical license, or, as I described what I thought that he deserved at the time in 2008, ‘having his medical license stripped from him, cut up in front of his face, and then the fragments ritually burned.” I stand by those words 15 years later.
Truth be told, I’ve been writing about Dr. Buttar intermittently since 2005, when I first discussed his use of chelation therapy to treat autism. As you might recall, chelation therapy uses what are called chelators, chemicals that bind to metals to better solubilize them and (supposedly) be excreted in the urine. As you might recall, one of the mainstays of autism biomed quackery back then was the use of chelation therapy to remove the mercury from the thimerosal preservative that had been in several childhood vaccines until around 2002 and was blamed for causing autism. Like many quack remedies, chelation therapy is an all-purpose treatment, falsely claimed to be useful for treating cardiovascular disease, Alzheimer’s disease, autism, and even cancer. It is also not a benign treatment. Calcium and magnesium are metals that can be chelated too. If their levels drop too low, it can cause cardiac arrhythmias and arrest, and autistic children have died as a result.
What made Dr. Buttar unique back then, so that he stood out from all the other antivax autism biomed quacks, was his “transdermal chelation therapy,” or, as we used to call it, Buttar’s butter. Of course, even if you accept that chelation therapy has any value whatsoever for treating anything other than acute toxicity due to excessive blood levels of certain heavy metals, there was no evidence that Buttar’s butter could even be absorbed through the skin, much less chelate anything. That was probably a good thing because, even if Buttar’s butter was utterly ineffective at treating autism, at least, unlike the case for intravenous chelation therapy, Buttar’s butter couldn’t kill anyone.
Those were seemingly simpler days. Dr. Buttar was definitely a cancer and autism quack, but he was also a significant “pioneer” of what became an all-too-common and distressing tactic among quacks since COVID-19: Attacking state medical boards and any other medical authority trying to rein in quacks and misinformation. His was the template that COVID-19 disinformation spreading doctors followed. Most people combatting disinformation now don’t appreciate just how influential Dr. Buttar was in terms of taking the war straight to state medical boards. The only one who did it more publicly was cancer quack Dr. Stanislaw Burzynski, but Dr. Rashid Buttar was the one who actually got North Carolina state law altered to make it more quack-friendly by mobilizing public support to neuter the North Carolina Medical Board and prevent it from disciplining doctors who practice quackery associated with “integrative medicine.” This is what I meant when I said that Dr. Buttar had basically won his battle with the North Carolina State Medical Board. He lost what was in retrospect a relatively minor battle, having been restricted from treating children and cancer patients, but went on to win the war by getting North Carolina law changed.
Those are some of the reasons why he was so widely admired among antivaxxers like Sayer Ji:
It is with sadness, yet great appreciation, that we pay tribute to my friend and colleague Dr. Rashid Buttar, who passed away on May 18th.
Rashid was a fearless and tireless advocate of health freedom and speaking truth to power, and he had an immense impact on helping the world understand the power of their bodies to heal, from literally anything.
We were less than a month awayfrom his 5th Annual Advanced Medicine Conference, which we are now uncertain will proceed.
Naturally, from an antivax perspective and given Dr. Buttar’s stature and relatively young age (57) at the time of his death, he couldn’t possibly have just died of heart disease. He had to have been killed.
“COVID-19 accines killed Dr. Buttar!” (Even though he was never vaccinated against COVID-19)
The conspiracy theories started flying fast and furious as soon as news of Dr. Buttar’s death was confirmed, starting with Sayer Ji himself in his tribute, starting with misrepresenting the Disinformation Dozen list as a “digital hit list”:
Given that Rashid was put on a digital hit list, along side me, cynically named the “disinformation dozen,” we had a special bond and interest in discussing the many ways we could continue to preserve health freedom, free speech, natural medicine, and ultimately bring the world into a healthier, happier, and more beautiful phase of evolution.
I will cherish these conversations, and Rashid’s incredibly sincere spirit, and know that while his life has been cut short, he helped make this world a safer, better place for us all through his good works.
Due to rumors circulating and the many questions I have received as to the cause of his passing, I wish to share the last publicly recorded discussion we had, where Rashid wanted the world to know the details surrounding the sudden decline of his health.
For the record, Rashid reached out to me on Feb. 18th, and explained that only a few weeks before, he was in the ICU for 6 days, with a diagnosis of both stroke and myocarditis, with symptoms and biomarkers consistent with adverse effects from the mRNA jabs (which he did not have). As you will see in the video, he believed that he was experiencing the result of shedding (aka, “self-amplifying” properties) from the transgenic mRNA jabs.
Rashid shares his story starting at the hour marker of an interview I conducted on March 13th with Dr. Joel Bohemier, Rashid, Ty Bollinger (another “hit list” member) and titled The Censorship Industrial Establishment Exposed.
Quack tycoon Joe Mercola’s girlfriend Erin Elizabeth was quick to add to the conspiracy mongering about Dr. Buttar’s death:
By the way he explains he was poisoned after the CNN interview. Rest in peace. https://t.co/97sZDYuGhs
— Erin Elizabeth Health Nut News 🙌 (@unhealthytruth) May 20, 2023
This is very consistent with what Erin Elizabeth does. She has long been peddling a conspiracy theory in which “They” (big pharma, of course) are hiring hit men to kill “holistic doctors” like Dr. Buttar for assassination, starting with Jeffrey Bradstreet (who killed himself), continuing to Nicholas Gonzalez (who died of an obvious heart attack), including other physicians who committed suicide, and ultimately totaling over 80 “holistic practitioners” whose deaths she blamed on pharma, whether the deaths were even plausibly suspicious. So of course she thinks that someone killed Dr. Buttar.
So did Dr. Paul Alexander, who wasted no time posting one of his long rambling Substack titles Was Buttar killed? ‘Dr Rashid Buttar died in mysterious circumstances just days after claiming he had been poisoned following a controversial interview with CNN about his Covid pandemic disinformation, in which he asked:
Covid conspiracy doctor claimed he was ‘poisoned after interview’ just days before death; IMO, I don’t put anything past the ‘dark unseen hand’, those behind COVID fraud; where is his autopsy?
Note the element added to the conspiracy theory, that Dr. Buttar had been “poisoned” after a CNN interview and that’s what accounted for his decline in health in 2023. Stew Peters, the most prominent purveyor of the “died suddenly” conspiracy theory claiming that COVID-19 vaccines are causing young healthy people—even elite athletes!—to drop dead suddenly due to blood clots, a message spread with his conspiracyfest of a movie disguised as a documentary, Died Suddenly, was quick out of the gate with his own take on the conspiracy theory that big pharma (or someone) “got” Dr. Buttar to “shut him up,” conspiracies that he promoted in this video three days after the announcement of Dr. Buttar’s death:
Right at the beginning of the video, Peters noted that Buttar was one of the believers in the “snake venom theory,” the “belief that COVID-19 was caused by snake venom rather than the virus” and that COVID-19 vaccines contain snake venom as well. You can almost hear the cognitive dissonance in his voice as he points out that the week before Buttar had “died suddenly,” adding that he knew that Dr. Buttar hadn’t been vaccinated and asking what could explain possibly explain his sudden death. And it is true that Buttar claimed that he had been “poisoned” by “200x what’s in the vaccine.” How this happened, he didn’t say. Peters then went on about the existence of CIA assassination techniques that make deaths look like heart attacks. Next up was an interview with Dr. Tau Braun, a quack whom I had never heard before (and whom I might have to look into more in the future), who repeated the narrative that whatever landed Dr. Buttar in the ICU for six days in January must have been poison and claiming that, when a man says he has been poisoned, he’s probably been poisoned. A South African, he referenced targeted assassinations during apartheid as having “shaped him” so that he believes that Buttar had been killed, whether by the CIA or someone else. He even repeated Sayer Ji’s claim that the Center for Countering Digital Hate, by placing Dr. Buttar on its Disinformation Dozen list, had actually put him on a hit list.
Again, because the false notion that healthy-appearing people (whether they were actually healthy, as Dr. Buttar appeared to be, or not, as Dr. Buttar was) almost never “died suddenly” before COVID-19 vaccines, Braun simply could not accept the possibility that Buttar had died of natural causes. So to Braun, Buttar must have been intentionally poisoned or, alternatively, “shedding” of spike protein (which is the same thing as the snake venom that COVID-19 supposedly is, according to this conspiracy theory) must have been the cause of his illness and demise. I’ve written about antivax resurrection of the old shedding trope to claim that those vaccinated with COVID-19 vaccines “shed” spike protein that can make others sick; more specifically, I’ve written about what utter nonsense the idea that the vaccinated “shed” COVID-19 spike protein is from a scientific standpoint. Braun even goes on to say that this toxin, conveniently enough for his narrative, is now “everywhere,” thanks to the vaccine, meaning that you can’t avoid it.
You know what does make a lot of spike protein? COVID-19 infection, that’s what!. It can also cause the body to release more virus (toxin, if you will, just to be sarcastic) that makes others sick. Of course, if this “toxin” is everywhere as Braun claims, then no one can avoid it and everyone needs quackery to “detox” themselves, including—surprise!—EDTA, which is a common chelator used by autism biomed quacks back in the day. At the end of the interview, Peters couldn’t resist interjecting that any doctors still giving COVID-19 shots are actively killing their patients, because of course he couldn’t. When Braun started claiming that the spike protein induces the body to produce “other toxins, and those toxins can leave the body and make other sick,” I almost lost it. Stochastic shedding, I guess.
In any event, you get the idea. Dr. Buttar had been antivaccine for decades before his death. I’ve been writing about his antivaccine quackery since 2005. So what really happened? I’ll try to integrate what happened from all the sources of which I’m aware.
Dr. Buttar: Ill since at least 2015
I’ll start at the beginning, using Jonathan Otto’s account, specifically the chapter in his e-book about Dr. Buttar’s story. By way of background, I knew this from accounts that I was able to read in the aftermath of Dr. Buttar’s death, before I learned of Otto’s ebook. Going back to the video referenced by Sayer Ji, Buttar noted that he had first felt ill in September 2022 at the Reawaken America tour, where he started feeling short of breath on stage, to the point where he couldn’t “properly give” his presentations and fell back on showing videos. Buttar then relates how he then also developed increasing swelling of his legs that got worse, lasted several days, resolved, and then came back. On Thanksgiving, he couldn’t get his boots on, and between Thanksgiving and January he put on 50 lbs of edema fluid. He said that by the time he was admitted to the hospital he couldn’t lie flat and still breathe and even had to stand up to breathe. These are absolutely classic symptoms of congestive heart failure, so much so that we teach them to first and second year medical students as part of cardiac physiology and pathophysiology.
Dr. Buttar was (ostensibly) a physician. He knew—or should have known—all of this. What was he doing during the months that he was putting on so much water weight and finding it increasingly difficult to breathe without being in an upright position? Why did he not see a doctor, particularly a cardiologist but at least an internist or family practice doc? He mentioned that he had thought about taking a diuretic, but apparently he never did. Did it not occur to him that his symptoms seemed consistent with right-sided heart failure, possibly biventricular failure? In any case, while he was in the ICU the doctors alternated between dosing him with Lasix and Bumex, two powerful diuretics, and got a lot of fluid off of him. He also then related having a mid-thalamic stroke confirmed by MRI that resolved. By the time he was released from the hospital, according to his account he had lost 60 lbs of fluid and then went on to lose nearly 20 lbs more after going home. At just before 1:09, he related having had elevated troponin levels, which he said were consistent with myocarditis and symptoms consistent with pericarditis. Ultimately, after his discharge from the ICU and hospital in January, Buttar blamed his illness on “secondary inoculation” by the COVID-19 vaccine shedding; indeed, he doubled down, claiming that the goal was to get half the population vaccinated, after which shedding would expose the rest of the population to the vaccines’s spike protein “toxin.” He concluded that he thinks “that’s what happened to me.” Because, of course it couldn’t possibly just have been heart disease.
Now add to the account above an early passage from his story in Otto’s ebook:
It was 6 days before I was scheduled to meet with Bryan, and I was getting ready to fly out to Spokane, Washington, for a speaking engagement. Actually, it was a Reawaken America tour. And I remember having a conversation with Clay Clark because the air quality there, normal is between 60 to a 100. As it gets up higher, it’s bad air quality.
A 100 is supposed to be the worst air quality. And the air quality in Spokane was 150. And I was having problems with breathing at that point, and I didn’t know what it was. I’ve got an underlying cardiac disposition, had a rapid CT angiography done back in 2015 that showed I had a 75 and a 95% occlusion, which didn’t make any sense because I’ve had all sorts of preventive treatments.
And the year before, I competed at the world championships in martial arts. So I’m competing at the world level in 2014 and 2015. I can’t walk a quarter of a mile without stopping 5 times because I’m getting short of breath and having chest pain. So I had this predisposition, and I’d been doing things to get that back, and I had. I was starting to sprint again.
Bryan is Bryan Ardis, a retired chiropractor and one of the purveyors of the conspiracy theory that COVID-19 is due to “envenomation,” not the SARS-CoV-2 virus, and that remdesivir is what kills COVID-19 patients, not COVID. Where did he get this idea?
Hilariously:
But the linchpin for his whole nonsensical theory is a 2017 episode of a network TV drama called “The Blacklist.”
After laying the groundwork for about half-an-hour, the video shows a clip from an episode of NBC’s “The Blacklist” in which the main character is poisoned with snake venom in a beverage.
“When I saw this, I knew,” Ardis says of watching the show, which originally aired in 2017 — almost three years before the outbreak of COVID-19. It’s now available on Netflix. “I knew I was right, I knew I was supposed to see that, because it was confirmation to me that other people knew this was planned all along, which, we’ve known this was a plan.”
One of the well-worn conspiracy theories that Ardis relies on here claims that the pandemic was planned in advance by nefarious actors — one of the most viral versions of this theory was presented in a pair of videos from 2020 called “Plandemic,” each of which we’ve written about.
As much as I’ve enjoyed James Spader’s scenery chewing portraying the fictional crime lord turned FBI informer Raymond “Red” Reddington in The Blacklist over the years, the show’s plotlines are often patently ridiculous (albeit in a very entertaining way), and obviously no source for anything resembling reliable medical information. That “Dr.” Ardis would admit that his inspiration for the idea that COVID-19 is “envenomation” tells you all you need to know about him.
Of course, Buttar’s account in Otto’s book sure doesn’t sound like that of a man who was the picture of health before he was exposed to the dreaded spike protein, which is ubiquitous and can’t be avoided thanks to the machinations of the evil big pharma cabal’s efforts to vaccinate the entire population, or who had been “poisoned” after an appearance on CNN several months ago. It sounds like a man in denial, who couldn’t believe that he had gone in a relatively short period of time from competing at a high level in the martial arts to not being able to walk a quarter of a mile without getting short of breath. He had clearly had significant heart disease diagnosed in 2015, four years before the pandemic and at least five years before COVID-19 vaccines! He admitted that he couldn’t walk more than a quarter of a mile without getting short of breath, which is not great. Even worse, though:
And so when this event was coming up, it was no different than the last 6 years. Since 2016, this has been going on. So it’s not like it’s anything new. But I was a little concerned because now the shortness of breath, over that last 6 years even though I had that cardiac history, over the last 4 years, I hadn’t had any shortness of breath or over the last 4 1/2 years, I hadn’t had any shortness of breath. I’d get the chest pain pretty regularly, but I wouldn’t get shortness of breath.
Now I was having shortness of breath. And I was having shortness of breath that was suffocating, where I felt like I was drowning. And it was slowly building up. And then I had this chronic cough. I had a rattling in my chest at nighttime. I felt like I was wheezing, but it wasn’t wheezing.
So anyway, there were all these little symptomologies that were going on. And so, I had my plane booked, and the next day I left to go there. And fortunately the event was outdoors. And fortunately, there was a nice wind blowing, and there was no issues with the quality of air that I felt, but I was short of breath when I was on stage. In fact, I even played more videos because I was a little short of breath.
To any physician, this sure sounds like an exacerbation of chronic heart disease and worsening congestive heart failure. What had Dr. Buttar been doing the last six years before the Reawaken America last September, given that by his own account he had had a critical occlusion of at least one of his coronary arteries, an occlusion that would normally mandate intervention? He’s fairly vague, saying only that he was “not oriented towards a pharmaceutical side.” Of course, CT-angiography is not the gold standard test to quantify the degree of coronary artery occlusion. However, in someone who is symptomatic—chest pain!—with such a test, further investigation would be mandated, and most likely cardiologists would recommend an angioplasty of the occluded vessels with stenting at minimum, a coronary artery bypass operation if too many vessels have a high degree of occlusion to stent. Again, Dr. Buttar was trained as a physician and was an ER doc for several years. He must have known this.
I facepalmed even more as I read how Dr. Buttar, experiencing not just chest pain but intermittent “suffocating” shortness of breath that made him feel as though he were “drowning,” dithered about whether to see a doctor and made plans to meet up with Ardis during the rest of his tour. Then, on the day he was going to see Ardis one afternoon when they were in the same city, this happened:
Well, I’ll tell you, when I walked out to the car to meet him, it was difficult for me to walk. And I felt like, to show you how serious it was for me, I felt like if I went to Bryan, I would be doing him a disservice because I was going to die in his house. And why would I do that, because then how is he going to be able to explain that? It would be a disservice for me to go there.
But I also knew if I went to a hospital, again, I was an ER doc for 7 years. I’m ACLS, ATTLS, Advanced Trauma Life Support, Advanced Cardiac Life Support, Pediatric Advanced Life Support, instructor certified. I teach doctors what to do and those things. And I knew that, if I go there, they’re not going to know what to do because I didn’t know what was going on. But I literally had a dilemma that made me stand outside the car trying to decide, do I go to the ER or do I go to Bryan?
It was like 2, 3 minutes, I just stood there. And I even remember thinking it was warm. There was a breeze blowing. I was right underneath a tree. The car’s parked. And it was a beautiful setting. And I just even thought maybe I should just go sit in the park because I really thought I was going to go. I had talked with my kids, I talked with Dr. Ashton, but never said anything to them. None of them knew. I made a call to an attorney to make sure that I had some semblance of a will, but I really felt that I was sliding. I could feel the life ebbing out of me. And I didn’t know why. I didn’t understand why.
So Dr. Buttar felt so weak and short of breath that he thought he’d die in Ardis’s house if he went there. In fact, he thought he was dying as he sat in his car, to the point where he made a call to his attorney to make sure his will was in order? I also couldn’t help but marvel at the cognitive dissonance. Buttar was ATLS and ACLS trained, instructor certified even? And he thought that ER docs wouldn’t know what to do with him if he showed up in their facility? Of course they would! From his own story, he almost certainly had severe congestive heart failure. They’d admit him to the ICU for supportive care, begin diuresing him to get the fluid off, evaluate his heart, and recommend further interventions.
He went to see Ardis instead. No, seriously. That’s what he did:
But he has said that to me since. And I’ve thought about those words since. And he said to me that there is no place on this planet that I would rather have you at right now than here in my living room. And he did some energetic testing. I won’t go into the details of all the stuff he did, but he did some energetic testing. And he told me that I had a poison in my heart. And I said, “A poison? You mean like a toxin?” He says, “No. A poison.”
And he told me at that time, he goes, “I don’t know what it is.” That’s what he told me. He goes, “But you’ve got a poison in your heart.” And then he told me, “There’s 4 chambers to the heart.” And so that everybody knows as the 2 atriums that are the receiving areas of the heart that receive the blood. And then the blood goes from those 2 atriums into the 2 ventricles, and the 2 ventricles pump the blood out to the rest of the body. And the 2 chambers that pump the blood out to the body are the ventricles. The ventricles are the right and left ventricle.
Ardis also told him:
And so the 2 atriums, on a scale of 0 to 12, the right atrium was functioning at a 3 on a scale of 0 to 12, with 12 being the best. And the left atrium was at a 4 on a scale of 0 to 12, with 12 being the best. And the ventricles, which are arguably the more important chambers of the heart, because they’re responsible for pumping blood either through the lungs or through the body, were both functioning at a 0 on a scale of 0 to 12, with 12 being the best. I mean, they weren’t functioning.
I was going to give Ardis credit for recognizing that Dr. Buttar was in a bad way until he claimed that his ventricles weren’t functioning at all. There’s a term for when the ventricles don’t function at all: Cardiac arrest. Dr. Buttar would not be conscious, much less talking, if his ventricles were truly “both functioning at a 0 on a scale of 0 to 12, with 12 being the best.” He’d be dead or on the way to being dead within minutes.
Thanks to Otto’s book, I also now know what Dr. Buttar was doing between the time he first noticed his “suffocating” shortness of breath at Reawaken America until he was admitted to the hospital for a six day ICU stay four months later: A whole lot of quackery. As he put on more and more edema fluid due to his congestive heart failure and possibly kidney failure, to the point where his feet “really ballooned, massively ballooned, became volleyball sized.” He even claimed that he couldn’t be having congestive heart failure because the edema was non-pitting. (Pitting edema is edema where, if you push the skin with a finger, will “pit”; i.e., leave an impression that takes a while to fill back in.) And while he’s correct that pitting edema is far more associated with congestive heart failure, in particular right-sided congestive heart failure, and non-pitting edema with other conditions, edema of this magnitude is a real problem. It could be lymphatic. It could be chronic venous insufficiency, which can cause impressive nonpitting edema. It could be myxedema from hypothyroidism.
Instead of going down the differential of pitting vs. nonpitting edema, putting it into context with all his other symptoms of congestive heart failure, and getting real medical care, Dr. Buttar listened to the chiroquack:
But by that time things were already better. So why was I having all this swelling in my feet? But think about it. If the body’s got a poison, if the heart’s trying to get rid of a poison, it’s going to try to pump it the furthest away from the heart that it possibly can, the furthest away from the brain. It’s going to try to shunt it to the furthest extremities it possibly can.
And so it sent it all to the most distal area of the body, which was the feet. And the kidneys couldn’t keep up with it. And because the kidneys, again, kidneys have their own, think of it as sensors or receptors. So if something’s going to come through the kidneys that’s going to damage the kidneys, the kidneys are going to slow down their functioning because they don’t want to get damaged. And they’re trying to handle the toxicity load or the poison load or whatever that’s trying to clear. But it’s also not going to handle more than what the kidneys can handle.
And so it was almost like it was pooling in my feet and in my legs and in my thighs because it was too much for the body to clear at that time. The kidneys couldn’t handle it, so it was clearing it as it could. And what’s interesting is that once it was cleared, I thought everything was fine.
In the meantime, he was doing “energetic healing,” medication, various woo prescribed to him by Ardis. He also underwent some sort of electrodermal testing that led him to be told:
And then I had another friend of mine, Dr. Group, had a test done for me and finds that I have a snake venom. I have a 2nd snake venom, I have a 3rd snake venom, I have an ant venom, and I have a spider venom. So it’s the polypus. I think that’s how you pronounce it. Spider, a wood ant venom, and then I had cobra venom, sorry, no, I had a rattlesnake venom and copperhead venom. And then later we found that I had cobra venom, too. But that’s what the test found. And I’m like, “What the hell? How did I get this stuff?”
He also noted how his symptoms were waxing and waning and that he was also taking a diuretic. He seems to have taken a diuretic for a while, until his symptoms got better, at which point he stopped it. Then—surprise! surprise!—eventually his symptoms would return.
I will say at this point that this interview recorded in Otto’s book must have occurred before mid-January, which is when Dr. Buttar wound up in the ICU, because no mention is made of that. He did, however, give a lengthy account of how he had come to conclude that, despite his never having been vaccinated against COVID-19, it must still have been the vaccines. First, he relates how he learned of 113 people who were having the same symptoms as he was (“shortness of breath, feeling of drowning, swelling in the legs, a dry, hacky cough”) but had not been vaccinated, which, among other things, convinced him that it must have been the shedding of spike protein from the vaccinated that had made him ill. (Never mind that he had had at the time a six or seven year history of diagnosed coronary artery disease.) As a result, Dr. Buttar concluded:
But it’s there, and it’s been orchestrated to be disseminated in a manner that is everywhere. There’s going to be no place that’s safe, unless you’re in a bunker underground 300 feet with an independent filtered water source and an independent filtered air source. It could be also be just through contact, and that’s been also postulated. Just things that we’re touching.
And:
People that have gotten exposed to shedding, which means that you don’t get the vaccine itself, but you’re in close proximity to other people that have gotten the so-called vaccine, the messenger RNA modified genetic experimentation that is introducing this spike protein into you. So people that have been close proximity with other people are experiencing symptomology that’s similar. And that’s because they’ve gotten what they call the shedding. The shedding is exposure of that spike protein that they have been exposed to. That’s exactly what happened to me because I haven’t had the jab, and I’d refused to take the jab, but I have had 2 specific exposures.
Which led Dr. Buttar to conclude:
And the 2nd time was when I gave a CNN interview, and they’d given me some water to drink. And I remember as I’m guzzling down the water, I’m thinking to myself, was that lid already open? I remember thinking I’m paranoid. And I got sick right after that; within half an hour I started getting sick.
So those are the 2 times that I could have been targeted or poisoned, if you will. Or it could have been just because I was sitting in a plane in close proximity to other people that had the vaccine or in some other type of public space. So I don’t know how I’d gotten it, but many people have experienced this viral shedding. And we call it viral shedding because nobody knew, really, what it was, but it’s actually shedding off this spike protein following the normal phenomena of viral shedding. But this is actually the spike protein that’s shedding, not a virus that’s shedding. And that spike protein is, again, that synthetic analog of these various types of poisons. Once they get into your system, then they are designed by design to elicit a response in the system. It’s targeting the heart cells, it’s targeting the cells of the reproductive system. So testicles in males and the ovaries in females, they have found an increased preponderance of the spike protein accumulating in those areas.
Sadly, a far more likely explanation for Dr. Buttar’s illness is that he had developed congestive heart failure, most likely from chronic ischemia due to cardiovascular disease, and that’s what caused him to develop the edema and difficulty breathing. He might also have had some venous insufficiency or other reason for his edema to be nonpitting. Who knows? Alternatively, maybe he had a silent myocardial infarction (or one with only mild symptoms that he ignored) and developed progressive heart failure afteward. He might even have had COVID-19, which caused a case of myocarditis. Again, who knows? The story as related by him and his followers lacks sufficient detail to allow for more than educated speculation about what happened to him. Basically, being in the ICU with congestive heart failure, for whatever reason, tells me that (1) you are not healthy, regardless of how healthy you think you are, and (2) you are at high risk of sudden cardiac death, which is very likely what happened to Dr. Buttar. Again, we can’t know for sure. Even adding the account in Otto’s book to the videos that I’ve examined, there is insufficient detail in all these accounts, even taken together, to make a more definitive assessment.
Quacks can be victims of their own quackery
The “died suddenly” conspiracy theory demands that healthy people cannot just “die suddenly” and that only the COVID-19 vaccine is causing people to “die suddenly” (even though there has been, for example, a small but consistent incidence of sudden arrhythmic deaths for a very long time, a phenomenon first described in the 1970s). When someone who is unvaccinated—particularly as famous an antivaxxer as Dr. Buttar—dies suddenly, antivaxxers cannot accept that it might have just been due to natural causes. It had to be due to something else, in this case COVID-19 vaccines. Of course, this presents a problem in the case of someone like Dr. Buttar, who because of his antivaccine views had never been vaccinated against COVID-19. Yet he died of what looks for all the world like heart disease. After all, a man who had been competed in martial arts and apparently been in robust health until 2014-2015, all while living the optimal healthy lifestyle and consuming the optimal healthy diet as an “integrative” and “natural” health practitioner, couldn’t possibly just develop heart disease in his early 50s, could he? No! That never happens, at least not according to the alternative medicine narrative. Unfortunately it can and, all too often, does happen. Diet and exercise decrease, but far from eliminate, one’s risk of developing heart disease.
Indeed, Dr. Buttar himself couldn’t accept that maybe he had just gotten sick, most likely as a result of genetics and/or bad luck. (Obviously, to him it couldn’t possibly have been due to lifestyle or diet, because as an “integrative medicine” physician he must have been living the ultimate healthy lifestyle.) Instead, he had to find a way to blame vaccine “shedding” for his illness. After all, he was unvaccinated and couldn’t possibly have had congestive heart failure and a small stroke without COVID-19 vaccines somehow having been responsible. In fairness, as far as I’ve been able to ascertain, before his death Dr. Buttar seemed not to believe that he had been “targeted” for assassination, but rather that he had been exposed to “shed” spike protein during the CNN interview or on an airplane. The even more outlandish conspiracy theories about a “hit” appear to have arisen more after his death. Because the antivaccine movement is at its heart a conspiracy theory like all science denial, people like Stew Peters and Erin Johnson were quick to spin conspiracy theories about the CIA and/or big pharma hit squads having “poisoned” him with the “toxin” from the vaccine. Of course, I have to ask here: If everyone who’s been vaccinated sheds spike protein, why would they ever have had to bother? No one ever said that conspiracy theories had to be consistent.
To conspiracy theorists like the late Dr. Buttar, the COVID-19 vaccine is the source of all evil, the weapon by which “They” are poisoning everyone for nefarious purposes of “control,” “depopulation,” or whatever combination of evil ends that the cranks dream up. Because he had kept himself “pure” by not getting the vaccine, Dr. Buttar simply could not accept getting seriously ill with heart disease for the same reasons that we are all prone to becoming seriously ill with heart disease. Consistent with the antivaccine movement, indeed the entire alternative medicine movement, being a cult of purity, he had to blame it on “contamination” or “poisoning” from an external source and toxin, the toxin being, of course, the mRNA-based COVID-19 vaccines, a conspiracy theory that he used before his sudden death to explain why he had been suffering chest pain and shortness of breath since 2016, the latter of which worsened in September 2022 to become “suffocating” shortness of breath. Then, during all that time, but particularly since September 2022, instead of seeking out science-based medicine to treat what was clearly worsening heart disease and bouts of congestive heart failure, Dr. Buttar relied on quacks like him (plus a little diuretic every now and then) to treat himself, even though just from what we know from his videos and Otto’s book about his longstanding heart disease and bouts of congestive heart failure, physicians can say that Dr. Buttar was at a high risk for sudden cardiac death, most likely due to a myocardial infarction.
Then, of course, after his death, Dr. Buttar’s fans had to find a way to turn his having “died suddenly” into not just an incidental exposure to “shed” spike protein but to a targeted assassination. Again, this is not unexpected. He also treated a life-threatening illness with a whole lot of the same quackery to which he had in practice subjected his patients over the last couple of decades. While it is true that he accepted science-based medicine (albeit, from his descriptions, with a number of caveats) in the ICU for six days in January, he appears not to have followed through afterward with a needed outpatient course of care to control his disease. If he had, he might well still be alive today. Antivax is, at its heart, a conspiracy theory, and grief will bring out its more paranoid elements. Unfortunately for him and his family, it turns out that Dr. Buttar was just as much a victim of his own quackery as his patients have been over the last quarter century.

Letter to a Medical Student: There Is No Elite RCT Strike Force
None of us have to fantasize what we would have done during a pandemic. What you actually did the past three years is exactly what you would have done.
Jonathan Howard on May 5, 2023
Link

Dear Mr. Knudsen,
I recently read your article on Sensible Medicine titled Death and Isolation During a Pandemic. You wrote movingly of your grandfather’s death and reflected on the tragedy of people dying alone at the start of the pandemic. As someone who lived through it all, I couldn’t agree more. Anyone who worked in the hospital at that time remembers patients suffering without their loved ones and aching goodbyes via IPads. I’m sure you’ve read about it. It happened to an overwhelming number of families who will be forever scarred. I’ll never forget one father who screamed at us over the phone that we didn’t try hard enough to save his 35-year-old son. Had he been there, he would have at least had the comfort of knowing we did everything possible. His son had clots throughout his body, and there was nothing to do. How horrible that he believes his son might still be alive had we only tried a bit harder.
However, your article contained this problematic section. You wrote:
Now, the empirical question, does limiting visitors reduce the transmission of SARS-CoV-2 and thus lower the incidence of COVID-19? The answer to this question is important and can provide a framework for thinking about the risk-benefit calculus of allowing visitors. Unfortunately, we don’t have an answer or any high quality data supporting this question, as shown in this systematic analysis. As discussed previously, there are a host of possible upsides to this policy; however, it is unclear if they are true.
Even with reasonable predictions, we live in a world that is messy and random, and thus require rigorously designed studies to uncover one form of the truth. I imagine that a cluster randomized controlled trial could have provided an answer. This would consist of randomizing hospitals into two groups: ones that allows visitors and others that restrict them. Then, over time, outcomes would be tracked in each group to measure the impact of the policy. I specifically left outcomes vague because the pertinent outcomes are debatable, but I would argue that a good starting point is COVID-19 hospital admissions. Another relevant outcome may be the number of positive COVID-19 cases in the surrounding community – although this would be more difficult to directly link to the policy. The broader point being that there are ways to uncover answers. There are smart people in positions of power with resources that can design and conduct such studies. Ultimately, they failed us, and we are left without answers and policies that caused significant harm.
Really, what would have been the point of your study?
It might not seem so, but there’s a lot to unpack here.
We both agree that randomized controlled trials (RCTs) are the “gold standard” in medicine, so how could I object to your call for a rigorous RCT to answer an important question? To answer that, I am going to do something that the doctors behind Sensible Medicine obviously didn’t do- I am going to respect you and take you seriously.
First of all, who exactly are “they“? Whom exactly did you have in mind when you wrote, “There are smart people in positions of power with resources that can design and conduct such studies“? You seem certain that someone had the time, resources, and authority to run your imagined RCT, but just chose not to for some stupid reason. Incredibly, you also think “they” should have had the foresight to run your imagined RCT. As morgues overflowed with victims of a new virus that lacked treatments and a vaccine, you think “they” should have been marshaling their resources to conduct an RCT of hospital visitation policies, as if there were no bigger fish to fry. Of course, your imagined RCT would have sounded utterly absurd at the start of the pandemic, which is why nobody actually suggested anything like it at the time.
You see, unless you know something I don’t, there isn’t a well-funded, elite RCT Strike Force that’s been training for a pandemic- though when a real virus arrived, they sat on the sidelines doing nothing for no particular reason. You seem to think that a large group of people with relevant medical expertise were really lazy. Who exactly do you think spent the pandemic twiddling their thumbs?
Maybe there should be an RCT Strike Force, but the cold truth is that no government or institution anywhere could have instantly whipped up dozens and dozens of rigorous RCTs to answer every pandemic conundrum and quandary, as you seem to imagine. In the real-world, politicians meddled with scientific agencies, and the CDC couldn’t even get basic tests right. Much of the federal response was in shambles from the pandemic’s earliest days. You should read about it. According to one recent article:
Early on, the government went off the rails.
In February 2020, the focus was on containment, with measures such as the travel ban on China and repatriation of Americans, including those stranded on cruise ships. Emergency mobilization efforts “languished,” the report noted. There was “confusion and friction about who was in charge of what problems.” The government’s “crisis action plan” amounted to little more than jargon. “There was little in it about what people would actually do.”
On Feb. 24, 2020, President Donald Trump tweeted from India, “The Coronavirus is very much under control in the USA … Stock market starting to look very good to me!” But according to the report, that same day, the White House task force concluded “containment was failing.” It was time to shift to mitigation. The next day, a high-ranking Centers for Disease Control and Prevention official, Nancy Messonnier, announced that community spread in the United States was inevitable. The stock market dived.
“President Trump was furious,” the report recalls. He kept downplaying the danger. “It’s going to disappear,” he said on Feb. 27. “Everything is really under control,” he said Feb. 29.
It was not. The authors of the report show, in detail, how federal crisis management “splintered by the third week of March.” HHS Secretary Alex Azar had placed the assistant secretary for preparedness and response, Robert Kadlec, in charge of the HHS effort — but at the same time, Vice President Mike Pence’s staff kicked him off the White House task force. The head of the Food and Drug Administration was not even on the task force for the first month. The CDC was “fractured into too many missions.” While some officials recognized the urgency of a crash program of testing and masks, “Kadlec had no money, no real emergency fund.”
“By late April, as a frightened and bewildered country became more and more confused about continuing business and school closures, and after some brow-raising comments at a White House briefing in which he discussed treating the virus with light, heat, or disinfectant, Trump essentially detached himself from his own government,” the report says. “He moved toward questioning and challenging what other government officials were doing.”
“The administration abdicated its wartime responsibility to lead,” they add. “It left the battlefield, and the war strategy” to the states and localities. By April, the White House chief of staff concluded the task force was “useless and broken.”
This is whom you imagine had the power, resources, and competency to have carried out your imagined RCT. They are “they.” 
Even if “they” had an elite RCT Strike Force at their disposal and bizarrely prioritized your imagined RCT at the start of the pandemic, it still faced many obstacles. I agree that COVID hospital admissions would be a reasonable outcome to track. But a woman who visits her sister at Hospital X, isn’t necessarily going to be admitted to Hospital X if she contracts COVID herself. At Bellevue, where I worked, nearly all of our patients were transferred from other overwhelmed hospitals. You should read about it. As such, your study would require that hospital visitors be followed and their hospital admissions tracked.
I am not an expert in this area, but it sure seems to me your imagined RCT would require informed consent for every hospital visitor. How would a grieving family feel if they lost another loved one and only later learned she was infected in a hospital that was part of a medical experiment to which she did not consent? Doesn’t she have the right to know the risks of your RCT, which include death, before she participates in it? At least one Twitter commentator had this to say about your proposed study:
Do you disagree? They ended their Twitter thread by saying, “The author of that blurb should have all their past contributions to field research reviewed for breaches of consent.”
Consenting every visitor at multiple hospitals while COVID raged would have been a Herculean task. Obtaining informed consent is a lengthy, formal process, not something done as people walk by you in the hallway. It’s not like asking people to sign a petition. Could you imagine actually trying to obtain informed consent from grieving families? What woman, on the verge of losing her husband of 50-years, would listen patiently as you explained the nature of your experiment? Much of this would have to be done via translator phones. Did you consider this? Many our patients did not speak English.
Tracking every visitor at multiple hospitals while COVID raged would also have been a Herculean task. How exactly do you think this should have been done? People would likely travel great distances to say goodbye to a loved one. Would grieving families be willing to share details of their fate to researchers over the telephone? It’s easy to imagine people would just want to be left alone if they got very sick with COVID as part of your experiment.Again, who exactly had the resources to do all this? Such expertise can’t be created out of thin air in a matter of weeks in the middle of a raging pandemic. There’s not an endless supply of clinical research coordinators. Why should these skilled professionals have dropped everything to run your imagined RCT? Thankfully, “they” chose to prioritize vaccine RCTs.
Many more questions and challenges remain:
Would administrators have let their hospitals participate in your imagined RCT? I doubt it. An RCT of hospital visiting policies wouldn’t have been on their radar at the pandemic’s start. They were too busy trying to get PPE, staffing, and places to store the dead bodies.
Would multiple hospital IRBs have approved a study that allowed large numbers of unvaccinated family members into unventilated rooms full of contagious patients? As morgues overflowed and forklifts were needed to move corpses into refrigerated trucks, would an IRB have actually accepted that there was true clinical equipoise and that your study needed to be run at that moment? I’m skeptical.
I work at two hospitals, NYU and Bellevue, that are blocks apart in terms of their distance, but worlds apart in terms of the patients they treat and communities they serve. To control for these discrepancies, you’d have to enroll a large number of hospitals. How many would have to participate to get meaningful results? What would happen to your study if some dropped out?
What if hospital staff objected to your study? We are packed like sardines on the elevators on a normal day. Your imagined RCT would have greatly increased our exposure to the virus at a time when no one was vaccinated. I wouldn’t have been thrilled to have been randomized to the “visitor’s allowed” hospital. We had enough sick healthcare workers as it was, and several of my co-workers died. You should read about them. Why didn’t you factor us into your imagined study?
What if other family members objected to your study? It’s easy to imagine family members strongly disagreeing about whether to visit a dying relative. Imagine an elderly couple living with their two adult children. If dad gets hospitalized, his immunocompromised daughter may strongly feel the risk of anyone visiting is too high. Does she get a say in your RCT? What if her brother visits their dad against her wishes and brings the virus home. What effect might this have on the family? He too might regret his decision, especially if anyone suffers as a result. What if your RCT created great conflict in people and families? Did you think about this at all?
What if the general public objected? Allowing hospital visitors would have meant more people on subways and buses, and therefore more risk for the essential workers who used them. You should read about their experience. Would these unvaccinated workers have had any input into your study?
Many hospitals were deluged with COVID patients in the spring of 2020. You should read about it. It’s not unreasonable to imagine that large groups of mourners in a hospital room would have greatly increased the spread of COVID in the spring of 2020. What contingency measures would be in place if that happened? Did you even consider that possibility?
Might patients and their families preferentially chose a hospital that was randomized to allow visitation? How would this affect the results?
Would a trial conducted in the spring of 2020 have any relevance today? I don’t think so. Both the population and the virus itself have changed too much. The original vaccine RCTs became obsolete in less than a year. Why would your RCT age better?
How long would your study have taken? In New York, one of the states with the strictest policies, hospital were mandated to allow visitors in June 2021. Do you really think your study would have been yielded actionable results before this? I doubt it.
Would you, or anyone else, support visitation restrictions today if your study found that hospital visitors increased the spread of COVID? Even RCTs that yield an unambiguous scientific result rarely allow us all to agree what policy should flow from them. They almost never provide a crystal clear answer to complicated social and ethical questions, a point you recognize. But have you given any thought into how your imagined RCT might actually inform policy? Let’s pretend your study found that allowing visitors increased community COVID infections by 14.3% (or any other number). Should hospitals allow visitors or not? Would everyone agree with your answer? Are you certain that your imagined RCT wouldn’t interject uncertainty and polarization into the rare issue where there seems to be near uniform agreement today? I’m not sure exactly when the policy of restricting visitors should have ended, but like everyone else, I’m glad it’s gone. So, even if your study were perfectly and rapidly executed at the pandemic’s start, for how long would it have held relevance? Everybody made up their mind about this issue long ago, and everyone agrees on it today. So really, what would have been the point of your study?
Challenge 1
Your imagined RCT didn’t occur to anybody at the start of the pandemic and there’s no one who actually could have run it had the idea oddly popped into their head. To be blunt, I think it’s just another sad example of An Impossible, Unethical Study of Dubious Value That’s Not Meant to be Taken Seriously. Since you disagree, I have two challenges for you- one for now and one for later.
My immediate challenge is to write the protocol for your imagined RCT. I challenge you write up the exact methods, outcome measures, and power analysis for an RCT that actually would have gotten IRB approval as well as buy-in from the relevant stakeholders and that actually would have been feasible at the pandemic’s start and actionable in a useful time frame.
For reference, the protocol for just one of the pediatric vaccine RCTs was 460 pages of dense policies and procedures. You should read it. I don’t expect you to produce such a document, but it shows the enormous thought and effort that serious people put into designing an RCT, even a completely straightforward one, when they actually intend for it to be done in the real world. Moreover, RCTs are very expensive and our public health agencies haven’t been drowning in money. Quite the opposite, actually. You should read about it. Your trial sounds expensive. So in your proposal, please include an estimate of how much your study will cost, where funding should come from, and how this might have impacted funding for RCTs for vaccines and COVID treatments.
Of course, writing a protocol and securing funding is the easy part. Once those are done, the real work begins. Research staff has to be hired and trained, hospitals have to agree to sign-up, multiple IRB boards have to give their approval, subjects have to be enrolled, outcomes have to be measured, and the data has to be analyzed and published. Running a large RCT is a logistical marvel with many moving parts and opportunities for things to go wrong. I suggest you listen to people who actually tried this pandemic. You could learn a lot. One researcher said:
I think of what my clinical research team went through to enroll people in that trial, and I thought my nurses were going to die. One of them got covid and got sick. Imagine trying to do that on a daily basis, multiple patients, some of them facing intubation, none of them have their families.
Another said:
There was one day our system had 84 deaths. And then you’re going to ask me to potentially put them on a placebo? It’s just really heart wrenching, talking with families, if your patients are able to communicate — and you’re dealing with all these deaths.
It turns out that actually running an RCT is a lot harder that just imagining one, especially during a pandemic. How would your study have overcome these obstacles?
Not surprisingly, graveyards are full of dead RCTs and there’s research into why trials fail. You should read about it. This continued during the pandemic. An article from the Gates Foundation titled “Uninformative Research” is the Global Health Crisis You’ve Never Heard Of said:
According to a 2021 study of thousands of COVID-19 drug trials, only 5% of the “study arms” included enough people to generate a statistically valid result. Chaos is to be expected during a crisis like a pandemic, but the norm during “peacetime” is nearly as worrisome. A recent study of U.S. clinical trials related to heart disease, diabetes, and lung cancer found that only 26% ended informatively. To put it bluntly: Three out of four were a waste of time and money.
Even trials “they” ran failed miserably during the pandemic. Although it was much less complicated that your imagined RCT, a proposed RCT of hydroxychloroquine and azithromycin run by the NIH closed in a month after enrolling just 20 patients, or just 1% of their goal. I’m sure that study looked great on paper. Why would your study succeed when “they” couldn’t enroll more than 20 patients in the most basic RCT imaginable?
Let’s see what you can come with.
Challenge 2
My second challenge for you is more long term. I challenge you to avoid a lamentable career of merely writing blogs, imaging one impossible RCT after another and then whining that “they” didn’t do the trials you imagined. Unfortunately, throughout the pandemic, a small number of doctors, who’ve yet to make any contribution to an actual RCT, have falsely claimed that RCTs and only RCTs can answer even the most basic questions and that dozens of complicated RCTs could have easily been done if only “they” weren’t so stupid and incompetent. These doctors never promoted an actual RCT or encouraged their audience to enroll in one. Like you, they pretended the RCT Strike Force was real, and rather than do the hard work of advancing an actual RCT, they merely fantasized aloud what they would have done if they were in charge of it. It’s sad that these doctors are more famous and influential than the largely anonymous researchers who actually ran RCTs, don’t you think? (You should read this too).
Such idle fantasizing about impossible RCTs is a waste of time for a highly-trained medical doctor. It’s just not a valuable contribution. Do you really want your memoir to be titled “RCTs I Imagined and Things I Would Have Done“? I hope not.
Instead, I challenge you not to sit on the sidelines, but rather to become “they” and actually do the RCTs you imagine. You wrote that “there are ways to uncover answers”, and you made it all sound so easy. Soon, you should be the one uncovering those answers with RCTs.
In the meantime, I suggest you use your voice to educate the public that RCTs aren’t the only means by which doctors can learn about the world. There haven’t been any RCTs of the virus after all, and your statement that visitor restrictions caused “significant harm” wasn’t based on an RCT, was it? You just knew.
We can also make important decisions without an RCT to guide us. No RCT proved it would be safe to allow visitors at hospitals. We just used common sense and collectively agreed that the harm of COVID patients dying alone exceeded the unknown risk of allowing visitors. This didn’t seem to bother you. Yet, if we had clinical equipoise at the start of the pandemic don’t we still have it today? After all, your imagined RCT wasn’t actually done. So, would you support doing your RCT today, even though it would mean some families would be barred from visiting their dying loved one? I doubt it. People rarely demand rigorous RCTs to justify policies they support.
I also suggest you also use your voice to educate the public on the value of RCTs. Being part of an experiment is scary for a lot of people, yet medicine couldn’t advance without willing volunteers. Being a “guinea pig” is a derisive term, when it should be a compliment. Though it’s not been the main focus of my career, I’ve tried to enroll patients in many different studies over the years. It’s not always easy. As you’ll find out, many patients don’t trust “they“. Unfortunately, articles like yours give the public every reason to distrust “they“- the actual people who actually try to enroll patients in RCTs. How do you think “they” feel when a medical student degrades them as you did? Do you think it makes their job easier? Do articles like yours actually advance the cause of RCTs?
I don’t think so.
And this upsets me, because I believe strongly in RCTs. Those of us who recognize we can’t study every aspect of the pandemic via an RCT are not against them in principle, as some doctors imply. It would be wonderful if running an RCT was as easy as imagining one. Because I want RCTs to actually succeed in the real-world I’ve encouraged people to enroll in them. Here’s what I said in the very first article I wrote this pandemic:
I’ve always believed that people who participate in medical research are doing a noble thing. If no one volunteered for such research, medicine would never advance.
I doubt that alone convinced anyone to enter an RCT, but I suspect if more doctors spread positive messages about RCTs, instead of berating “they“, this would have had a positive impact in the real-world. My dedication is such that I volunteered to be in a vaccine-RCT. My contribution was a small one,- the study took place at my hospital and I got snacks- but at least it was a real one. I want my actions to match my words, and I don’t feel like a complete backseat driver when I say that more RCTs should have been done this pandemic. The UK’s RECOVERY program was a model for what should have been, and I actually think a well-funded, elite RCT Strike Force would be amazing.
Until that exists, however, your statement that there are “smart people in positions of power with resources that can design and conduct such studies” was gross misinformation. Endless calls for someone else to run impossible RTCs of everything are just a way for the commentariat class to do what they’ve done the entire pandemic; spread fear, uncertainty, and doubt about common sense measures to control a rapidly-spreading virus in a world with finite time and resources. Doctors imagine impossible studies not because they actually want them to get done, but rather to sow doubt in the public’s mind. They want people to think that we shouldn’t do anything until we know everything. None of this is new. Casting doubt on vaccines by calling for an impossible RCT of the entire vaccine schedule over the entire human lifespan looking at every outcome imaginable is one of the oldest anti-vaxxer tricks in the book. You should read about it.
Fantasies of impossible studies always end with some version of what you said: “they failed us, and we are left without answers and policies that caused significant harm.” In the real-world, during a raging pandemic with a brand new virus, “they” had to make many difficult decisions without the benefit of a rigorous RCT to guide them at every turn. Yet, doctors who merely imagine RCTs absurdly claim “they” have only themselves to blame for their predicament.
Unfortunately, this was all part of a larger trend where doctors with no real-world responsibility for anything made impossible things sound very easy in Tweets, editorials, and YouTube videos. They acted as if there were no difference between writing “protect the vulnerable” and actually protecting the vulnerable. As I wrote previously:
In the imaginary world, all it took to protect nursing homes and open schools was to say “protect nursing homes” and “open schools”. In the real world, this was impossible when the virus was allowed to spread freely.
When impossible things could not be done effortlessly in the real-world, “they” were always to blame. Is it any surprise that in the real-world, public health officials were routinely threatened and even attacked by angry, paranoid citizens? You should read about it.
There’s nothing inherently wrong with fantasizing how you would have handled the pandemic had you been given unlimited time, money, and power, as well as total public cooperation with all of your imagined plans- basically God-like powers. However, you should recognize that this is just play-acting, not a serious contribution. You see, none of us have to fantasize what we would have done during a pandemic. What you actually did the past three years is exactly what you would have done.
And something tells me that if you’re brave enough to devote your career to actually running RCTs, you’ll no longer be impressed by doctors who merely imagine RCTs they would have done, while moaning “they failed us” because in the real world, actually running an RCT is a lot harder than just imagining one.
Note: I also turned this into a video essay.

Critical Thinking in Medicine
Cognitive Errors and Diagnostic Mistakes is a superb new guide to critical thinking in medicine written by Jonathan Howard. It explains how our psychological foibles regularly bias and betray us, leading to diagnostic mistakes. Learning critical thinking skills is essential but difficult. Every known cognitive error is illustrated with memorable patient stories.
Harriet Hall on January 15, 2019
Link


Critical thinking in medicine is what the Science-Based Medicine (SBM) blog is all about. Jonathan Howard has written a superb book, Cognitive Errors and Diagnostic Mistakes: A Case-Based Guide to Critical Thinking in Medicine, that epitomizes the message of SBM. In fact, in the Acknowledgements, he credits the entire team at SBM for teaching him “an enormous amount about skepticism and critical thinking”, and he specifically thanks Steven Novella, Harriet Hall (moi!), and David Gorski.
Dr. Howard is a neurologist and psychiatrist at NYU and Bellevue Hospital. The book is a passionate defense of science and a devastating critique of Complementary and Alternative Medicine (CAM) and pseudoscience. Its case-based approach is a stroke of genius. We humans are story-tellers; we are far more impressed by stories than by studies or by textbook definitions of a disease. Dr. Howard points out that “Anecdotes are part of the very cognition that allows us to derive meaning from experience and turn noise into signal.” They are incredibly powerful from an emotional standpoint. That’s why he chose to begin every discussion of a cognitive error with a patient’s case, an anecdote.
CAM knows how effective this can be; that’s why it relies so heavily on anecdotes. When doctors think of a disease, they are likely to think of a memorable patient they treated with that disease, and that patient’s case is likely to bias their thinking about other patients with the same disease. If there is a bad outcome with a treatment, they will remember that and may reject that treatment for the next patient even if it is the most appropriate one. Dr. Howard uses patient stories to great advantage, first providing the bare facts of the case and then letting the patient’s doctors explain their thought processes so we can understand exactly where and why they went wrong. Then he goes on to explain the psychology behind the cognitive error, with study findings, other examples, and plentiful references. If readers remember these cases, they might avoid similar mishaps.
An encyclopedia of cognitive errors
The book is encyclopedic, running to 30 chapters and 588 pages. I can’t think of anything he failed to mention, and whenever an example or a quotation occurred to me, he had thought of it first and included it in the text. I couldn’t begin to list all the cognitive errors he covers, but they fall roughly into these six categories:
    • Errors of overattachment to a particular diagnosis
    • Errors due to failure to consider alternative diagnoses.
    • Errors due to inheriting someone else’s thinking.
    • Errors in prevalence perception or estimation.
    • Errors involving patient characteristics or presentation context.
    • Errors associated with physician affect, personality, or decision style.
A smattering of examples
There is so much information and wisdom in this book! I’ll try to whet your appetite with a few excerpts that particularly struck me.
    • Discussing an issue with others who disagree can help us avoid confirmation bias and groupthink.
    • Negative panic: when a group of people witness an emergency and fail to respond, thinking someone else will.
    • Reactance bias: doctors who object to conventional practices and want to feel independent may reject science and embrace pseudoscience.
    • Cyberchondria: using the Internet to interpret mundane symptoms as dire diagnoses.
    • Motivated reasoning: People who “know” they have chronic Lyme disease will fail to believe 10 negative Lyme tests in a row and then believe the 11th test if it is positive.
    • The backfire effect: “encountering contradictory information can have the paradoxical effect of strengthening our initial belief rather than causing us to question it.”
    • Biases are easy to see in others but nearly impossible to detect in oneself.
    • Checklists for fake diseases take advantage of the Forer effect. As with horoscopes and cold readings, vague, nonspecific statements convince people that a specific truth about them is being revealed. Fake diseases are unfalsifiable: there is no way to rule them out.
    • When presenting risk/benefit data to patients, don’t present risk data first; it will act as an “anchor” to make them fixate on risk.
    • The doctor’s opinion of the patient will affect the quality of care.
    • Randomness is difficult to grasp. The hot hand and the gambler’s fallacy can both fool doctors. If the last two patients had disease X and this patient has similar symptoms, the doctor will think he probably has disease X too. Or if the doctor has just seen two cases of a rare disease, it will seem unlikely that the next patient with similar symptoms will have it too.
    • Apophenia: the tendency to perceive meaningful patterns with random information, like seeing the face on Mars.
    • Information bias: doctors tend to think the more information, the better. But tests are indicated only if they will help establish a diagnosis or alter management. They should not be ordered out of curiosity or to make the clinician feel better. Sometimes doctors don’t know what to do with the information from a test. This should be a lesson for doctors who practice so-called functional medicine: they order all kinds of nonstandard tests whose questionable results give no evidence-based guidance for treating the patient. Doctors should ask “How will this test alter my management?” and if they can’t answer, they shouldn’t order the test.
    • Once a treatment is started, it can be exceedingly difficult to stop. A study showed that 58% of medications could be stopped in elderly patients and only 2% had to be re-started.
    • Doctors feel obligated to “do something” for the patient, but sometimes the best course is to do nothing. “Just don’t do something, stand there.” At the end of their own life, 90% of doctors would refuse the treatments they routinely give to patients with terminal illnesses.
    • Incidentalomas: when a test reveals an unsuspected finding, it’s important to remember that abnormality doesn’t necessarily mean pathology or require treatment.
    • Fear of possible unknown long-term consequences may lead doctors to reject a treatment, but that should be weighed carefully against the well-known consequences of the disease itself.
    • It’s good for doctors to inform patients and let them participate in decisions, but too much information can overwhelm patients. He gives the example of a patient with multiple sclerosis whose doctor describes the effectiveness and risks of 8 injectables, 3 pills, and 4 infusions. The patient can’t choose; she misses the follow-up appointment and returns a year later with visual loss that might have been prevented.
    • Most patients don’t benefit from drugs; the NNT tells us the Number of patients who will Need to be Treated for one person to benefit.
    • Overconfidence bias: in the Dunning-Kruger effect, people think they know more than the experts about things like climate change, vaccines and evolution. Yet somehow these same people never question that experts know how to predict eclipses.
    • Patient satisfaction does not measure effectiveness of treatment. A study showed that the most satisfied patients were 12% more likely to be admitted to the hospital, had 9% higher prescription costs, and were 26% more likely to die.
    • The availability heuristic and the frequency illusion: “Clinicians should be aware that their experience is distorted by recent or memorable [cases], the experiences of their colleagues, and the news.” He repeats Mark Crislip’s aphorism that the three most dangerous words in medicine are “in my experience”.
    • Illusory truth: people are likely to believe a statement simply because they have heard it many times.
    • What makes an effective screening test? He covers concepts like lead time bias, length bias, and selection bias. Screening tests may do more harm than good. The PSA test is hardly better than a coin toss.
    • Blind spot bias: Everyone has blind spots; we recognize them in others but can’t see our own. Most doctors believe they won’t be influenced by gifts from drug companies, but they believe others are unconsciously biased by such gifts. Books like this can make things worse: they give us false confidence. “Being inclined to think that you can avoid a bias because you [are] aware of it is a bias in itself.”
    • He quotes from Contrived Platitudes: “Everything happens for a reason except when it doesn’t. But even then you can in hindsight fabricate a reason that will satisfy your belief system.” This is the essence of what CAM does, especially the versions that attribute all diseases to a single cause.
Some juicy quotes
Knowledge of bias should contribute to your humility, not your confidence.
Only by studying treatments in large, randomized, blinded, controlled trials can the efficacy of a treatment truly be measured.
When beliefs are based in emotion, facts alone stand little chance.
CAM, when not outright fraudulent, is nothing more than the triumph of cognitive biases over rationality and science.
Reason evolved primarily to win arguments, not to solve problems.
More
He includes a thorough discussion of the pros and cons of limiting doctors’ work hours, with factors most people have never considered, and a thorough discussion of financial motivations.
The book is profusely illustrated with pictures, diagrams, posters, and images from the Internet like “The Red Flags of Quackery” from sci-ence.org. Many famous quotations are presented with pictures of the person quoted, like Christopher Hitchens and his “What can be asserted without evidence can be dismissed without evidence”.
He never goes beyond the evidence. Rather than just giving study results, he tells the reader when other researchers have failed to replicate the findings.
We rely on scientific evidence, but researchers are not immune from bias. He describes the many ways research can go astray: 235 biases have been identified that can lead to erroneous results. As Ioannidis said, most published research findings are wrong. But all is not lost: people who understand statistics and the methodologies of science can usually distinguish a good study from a bad one.
He tells the infamous N-ray story. He covers the file drawer effect, publication bias, conflicts of interest, predatory journals, ghostwriting, citation plagiarism, retractions, measuring poor surrogates instead of meaningful clinical outcomes, and outright fraud. Andrew Wakefield features prominently. Dr. Howard’s discussions of p-hacking, multiple variables, random chance, and effect size are particularly valuable. HARKing is Hypothesizing After the Results are Known. It can be exploited to create erroneous results.
He tells a funny story that was new to me. Two scientists wrote a paper consisting entirely of the repeated sentence “Get me off your fucking mailing list” complete with diagrams of that sentence. It was rated as excellent and was accepted for publication!
Conclusion: Well worth reading for doctors and for everyone else
As the book explains, “The brain is a self-affirming spin-doctor with a bottomless bag of tricks…” Our brains are “pattern-seeking machines that fill in the gaps in our perception and knowledge consistent with our expectations, beliefs, and wishes”. This book is a textbook explaining our cognitive errors. Its theme is medicine but the same errors occur everywhere. We all need to understand our psychological foibles in order to think clearly about every aspect of our lives and to make the best decisions. Every doctor would benefit from reading this book, and I wish it could be required reading in medical schools. I wish everyone who considers trying CAM would read it first. I wish patients would ask doctors to explain why they ordered a test.
The book is not inexpensive. The price on Amazon is $56.99 for both softcover and Kindle versions. But it might be a good investment: you might save much more money that that by applying the principles it teaches, and critical thinking skills might even save your life. Well-written, important, timely, easy, and entertaining to read, lots of illustrations, packed with good stuff. Highly recommended.

Post Hoc Bias in Medicine
We are more likely to use an ineffective treatment after slight symptom improvement, even when the changes are coincidental.
Scott Gavura on September 26, 2024
Link
Casey, our Labrador Retriever, started limping before she was a year old. Investigations confirmed dysplasia, a genetic condition that leads to degenerative joints, arthritis, and pain. We were devastated. After considering the few treatment options that existed, we decided to skip surgery and treat it conservatively. I had no desire to start anti-inflammatory drugs, being acutely aware of the side effects from chronic use. I was familiar with a popular supplement with weak but somewhat promising evidence: We started giving her glucosamine and chondroitin supplements regularly. And we watched and waited.
It took some time, but Casey did appear to improve. We were thrilled. Life went on, and other than the occasional rough play session, Casey’s limping was mild, and she thrived. We continued the supplements, confident that we were doing good. But eventually I started paying attention to the emerging evidence on glucosamine and chondroitin. Once touted as a panacea for arthritis and joint pain, there had finally been some high-quality trials conducted – and the results were disappointing. I started to wonder if the supplements were really doing anything for my dog’s pain. Eventually I decided on a trial – so I stopped the supplements about seven years after I started them. Neither my wife nor I could notice any difference at all in her mobility. Nor did the veterinarian. We’d been fooling ourselves, spending hundreds of dollars in the process.
Post hoc ergo propter hoc (after this, therefore because of this)
My experiment with glucosamine and chondroitin was definitely not the only time I’ve been subject to post hoc bias, but it’s the one that still stands out to me. Sometimes called the post hoc ergo propter hoc fallacy, it attaches causality to a sequence of events.
Vaccine fears have been driven by the post hoc ergo propter hoc fallacy. The early signs of autism may be observed to emerge at the same time children receive many vaccinations – in the first few years of life. This led some people to incorrectly infer a causal link between the two, simply based on the timing of events.
It’s not always harms where this fallacy may be observed. Just because your condition improves after a treatment, it doesn’t mean the treatment caused the improvement. Other possible factors could be at work, including mean reversion, time and natural healing, the placebo effect, or simply chance. Attributing an improvement to a treatment is often comforting for patients, easily explained to families, and convenient for clinicians. It’s also a quick and common explanation, that may be wrong.
Much of medicine is still filled with uncertainty. We don’t fully understand all disease mechanisms. We don’t always know why some drugs work in some patients and not others. If we’re not cautious about rigorously testing our assumptions, this logical fallacy can lead to wrong diagnoses, premature conclusions, overtreatment, and wasted resources.
Tunnel vision decision-making
Post hoc bias has not been not widely studied in medicine. In a paper published in September 2024 in JAMA Network Open, Donald Redelmeier and Eldar Shafir examined judgements in health treatments for signs of post hoc bias. Their hypothesis was that people would be more likely to continue treatments, even questionable ones, if they observed minor improvements in symptoms after initial care.
The researchers created different case scenarios about a patient with vague symptoms of unclear severity. Each case aimed to gather a treatment decision for a hypothetical patient by asking, “Would you recommend that [patient] continue or discontinue [treatment]?” Participants could choose from four options: “definitely continue,” “tend to continue,” “tend to discontinue,” or “definitely discontinue.”
Each scenario had two versions: an “improved version” where symptoms slightly improved after treatment, and an “unchanged version” where symptoms stayed the same. No scenario involved worsening symptoms that would suggest stopping the treatment. Otherwise, both versions were identical, involving the same patient and single question, and were randomly assigned.
Two different groups were surveyed: potential patients and healthcare professionals who regularly interact with patients. The largest group was public members recruited through the Prolific Survey platform. The second group included active community pharmacists who were approached in person at their workplaces during off-peak hours.
All participants completed the survey anonymously, were unaware of the study’s hypothesis, and only saw one scenario without knowing about alternative versions. Community participants took the survey online and were compensated. Pharmacists were approached with the script, “I would like to talk with a pharmacist,” and additional replies were prepared using “secret shopper” methods.
The Scenarios and the responses
Antibiotics
The first case involved using antibiotics against medical advice. The improved and unchanged versions were the same, except for one sentence in bold:
“JL is a 35-year-old schoolteacher. She has a sore throat and starts an antibiotic treatment from an unused prescription offered by a friend. According to medical science, however, misusing antibiotics might eventually cause resistant organisms. The next day JL feels better [unchanged]. Would you recommend she continue or discontinue the antibiotic?”
300 people participated—150 saw the improved version, and 150 saw the unchanged version. More people recommended continuing the antibiotic when JL felt better than when symptoms were unchanged (67 people [45%] vs. 25 people [17%]). This showed a moderate increase in support for continuing treatment (OR, 3.98; 95% CI, 2.33-6.78; P < .001).
Sugar supplement
This scenario involved a sugar supplement for insomnia: Only the bold text changed:
“GD is a 45-year-old administrator. She has irregular insomnia and starts a sugar powder supplement in the hopes of getting better sleep. The next week her sleep is better [unchanged]. Would you recommend she continue or discontinue the sugar powder supplement?”
200 people participated, divided equally between the two scenarios. More recommended continuing the sugar when symptoms improved (83%) (vs. unchanged (17%)). This showed a strong increase in support for continuing treatment (OR, 22.77; 95% CI, 10.99-47.18; P < .001).
Acupuncture
This scenario involved a family member suggesting acupuncture to a young woman with neck pain. Only the bold text differed:
KA is a 30-year-old flight attendant. She feels intermittent neck pain and starts acupuncture for relief (suggested by her grandmother). The next week she feels better [unchanged]. Would you recommend she continue or discontinue the acupuncture?”
198 individuals responded: 100 received the “improved” version and 98 the “unchanged” version. More recommended continuing the acupuncture if symptoms were reported to improve [87%] than when symptoms were unchanged [28%]. (OR, 15.27; 95% CI, 7.46-31.27; P < .001)
Copper Bracelet
This scenario involved a bracelet advertised and promoted for wrist pain. Only the bold text differed:
“MB is a 45-year-old accountant. She feels intermittent wrist pain and starts wearing a copper bracelet for pain relief (advertised in a magazine). The next week she feels better [unchanged]. Would you recommend she continue or discontinue the copper bracelet?”
199 individuals responded: 100 received the “improved” version and 99 the “unchanged” version. More recommended continuing using the copper bracelet after symptoms were reported to improve [78%], than when symptoms were unchanged [25%]. (OR, 16.19; 95% CI, 5.32-19.52; P < .001)
Shampoo Scenario
This scenario involved “horse shampoo” and baldness – about as ridiculous as you can get:
“DK is a 75-year-old retired professor. He is going bald and starts applying horse shampoo once per day. However, researchers typically agree horse shampoo does not promote hair growth. A couple of weeks later his hair seems better [unchanged]. Suppose he is undecided and asks you, would you recommend DK continue or discontinue the horse shampoo?”
400 individuals responded: 200 received the improved version and 200 the unchanged version. More recommended continuing the horse shampoo after symptoms improved [65%] than when symptoms were unchanged [7%] (OR, 23.30; 95% CI, 12.69-42.16; P < .001)
Vitamin Supplements: Public Responses
This scenario involved the use of vitamins for fatigue. In the absence of a deficiency, there is no evidence to support the use of vitamins to treat fatigue. Only the bold text differed:
“WS is a 30-year-old hospital nurse. He feels tired and starts a Vitamin B12 supplement for energy (despite normal B12 levels). The next week he feels better [unchanged]. Would you recommend he continue or discontinue the Vitamin B12 supplement?”
200 individuals responded: 100 received the improved version and 100 the unchanged version. Despite a normal baseline B12 level more recommended continuing after symptoms improved [80%] than when symptoms were unchanged [33%] (OR, 7.91; 95% CI, 4.18-14.97; P < .001)
Vitamin Supplements: Pharmacist Responses
This scenario involved pharmacists and a “secret shopper” in pharmacies:
“Hi, I am wondering if I can talk with a pharmacist? [Wait for pharmacist.] Last week I was feeling tired and started taking a Vitamin B12 supplement for energy. Now this week I feel better [unchanged]. Do you think I should continue or discontinue it?”
100 pharmacists were consulted, of whom 44 and 56 received the improved and unchanged versions, respectively. More pharmacists recommended continuing the treatment after symptoms improved [82%] than when the symptoms were unchanged [63%]. (OR, 2.60; 95% CI, 1.04-6.52; P = .03)
This is disappointing, as pharmacists were more likely than the general public to recommend continuing vitamin B12 in the absence of any persuasive reason for use.
Source: xkcd.com
Conclusion: Beware Post Hoc Bias
This survey study revealed a consistent pattern of post hoc bias, where the public (and in one scenario, pharmacists) recommended continuing a treatment if they experienced or identified some subjective improvement, regardless of whether the treatment may or may not have caused it. As demonstrated by these often ridiculous examples, post hoc bias may lead patients and health professionals to persist with treatments that may be having no meaningful effects. The authors make an important observation that’s worth quoting in full:
Clinicians often acknowledge complexity in the aftermath of failure – namely, patient outcomes are uncertain. That same humility can feel less likely in the aftermath of a success where adherence to scientific discipline can be difficult following a treatment initiated by the responsible clinician. Personal motivations might further reinforce undue confidence after an improvement since clinicians want to believe their work makes a positive difference. Faulty attribution may be further exacerbated when the clinician interacts with a grateful patient, appreciative family, and impressed colleagues. In short, the system surrounding care helps perpetuate the post hoc bias.
Post hoc bias may feel intuitive, but it’s not necessarily accurate. While post hoc reasoning makes sense when the diagnosis is clear and treatment effects are well known, it may lead to complacency and not seeking effective treatments – one of my primary criticisms of alternative medicine. Recognizing this bias is a key first step to minimizing its influence.

Are medical errors really the third most common cause of death in the U.S.?
A regurgitation of existing data suggested that medical error is the third leading cause of death in America. Is it true? Spoiler alert!
No. No it’s not. While medical error can and should be reduced, this BMJ article does not justify claims that doctors are a leading cause of death in the United States.
David Gorski onMay 9, 2016
Link

NOTE: Anyone who has seen several derogatory articles about me on the web and is curious about what the real story is, please read this and this.

It is an unquestioned belief among believers in alternative medicine and even just among many people who do not trust conventional medicine that conventional medicine kills. Not only does exaggerating the number of people who die due to medical complications or errors fit in with the world view of people like Mike Adams and Joe Mercola, but it’s good for business. After all, if conventional medicine is as dangerous as claimed, then alternative medicine starts looking better in comparison.
In contrast, real physicians and real medical scientists are very much interested in making medicine safer and more efficacious. One way we work to achieve that end is by using science to learn more about disease and develop new treatments that are as efficacious or more so than existing treatments with fewer adverse reactions (clinical equipoise). Another strategy is to use what we know to develop quality metrics against which we measure our practice. Indeed, I am heavily involved in just such an effort for breast cancer patients. Then, of course, we try to estimate how frequent medical errors are and how often they cause harm or even death. All of these efforts are very difficult, of course, but perhaps the most difficult of all is the last one. Estimates of medical errors depend very much on how medical errors are defined, and whether a given death can be attributed to a medical error depends very much on how it is determined whether a death was preventable and whether a given medical error led to that death.
Some cases are easy. For example, if I as a surgeon operating in the abdomen were to slip and put a hole in the aorta, leading to the rapid exsanguination of the patient, it’s obvious that the error caused the patient’s death. But what about giving the wrong antibiotic in a septic patient who is critically ill with multisystem organ dysfunction? Other examples come to mind. When should a given medical error in such a critically ill patient who has a high probability of dying even with perfect care be blamed if that patient dies? It’s not a straightforward question.
Perhaps that’s why estimates of the number of deaths due to medical error tend to be all over the map. Consider this. According to the CDC, there are approximately 2.6 million deaths from all causes in the US every year. Now consider these headlines from last week:
    • “Medical Errors Are No. 3 Cause Of U.S Deaths, Researchers Say (NPR)”
    • “Researchers: Medical errors now third leading cause of death in United States” (The Washington Post)
    • “Why Are Medical Mistakes Our Third Leading Cause of Death?” (Huffington Post)
    • “Second study says medical errors third-leading cause of death in U.S.” (USA TODAY)
These stories all refer to an article last week in BMJ by Martin A Makary and Michael Daniel entitled “Medical error—the third leading cause of death in the US,” which claims that over 251,000 people die in hospitals as a result of medical errors. Given that, according to the CDC, only 715,000 of those deaths occur in hospitals, if Makary and Daniel’s numbers are to be believed, some 35% of inpatient deaths are due to medical errors. That’s just one reason why there are a lot of problems with this article, but there are even more problems with how the results have been reported in the press and the recommendations made by the authors.
Error inflation?
The first thing I noticed that surprised me about this BMJ article is that it isn’t a fresh study at all. Rather it’s a regurgitation of already existing data. It is not a “second study,” as, for example, the USA TODAY headline calls it. It’s just a pooling of existing data to produce a point estimate of the death rate among hospitalized patients reported in the literature extrapolated to the reported number of patients hospitalized in 2013 based on four major existing studies since the Institute of Medicine (IOM) report “To Err Is Human” in 1999. Basically, it’s not unlike a similar pooling and extrapolation of studies performed by John James in 2013. Yet, in every article I’ve seen about it, it’s described as a study. In reality, it’s more an op-ed calling for better reporting of deaths from medical errors, with extrapolations based on studies with small numbers. That’s not to denigrate the article just for that. Such analyses are often useful; rather it’s to point out how poorly this article has been reported and how few seemed to notice that this article adds exactly nothing new to the primary scientific literature.
Here’s a passage from Dr. Makary’s article that disturbed me right off the bat:
The role of error can be complex. While many errors are non-consequential, an error can end the life of someone with a long life expectancy or accelerate an imminent death. The case in the box shows how error can contribute to death. Moving away from a requirement that only reasons for death with an ICD code can be used on death certificates could better inform healthcare research and awareness priorities.
Case history: role of medical error in patient death
A young woman recovered well after a successful transplant operation. However, she was readmitted for non-specific complaints that were evaluated with extensive tests, some of which were unnecessary, including a pericardiocentesis. She was discharged but came back to the hospital days later with intra-abdominal hemorrhage and cardiopulmonary arrest. An autopsy revealed that the needle inserted during the pericardiocentesis grazed the liver causing a pseudoaneurysm that resulted in subsequent rupture and death. The death certificate listed the cause of death as cardiovascular.
This example is nowhere near as straightforward an example as the authors appear to think it is. In fact, it’s an utterly horrible example. For one thing, notice the weasel wording. We’re told that the the patient was evaluated with “extensive tests, some of which were unnecessary, including a pericardiocentesis.” This implies that the pericardiocentesis wasn’t necessary, but an equally valid interpretation is that it was just one of the tests. Besides, without a lot more details, it’s impossible to tell whether the pericardiocentesis was necessary or not. While the point that the percutaneous procedure contributed to this patient’s death is valid, how do we classify this? Delayed bleeding is a known complication of percutaneous procedures, as is damage to adjacent organs that are potentially in the path of the needle. Even when done by expert hands, such procedures will cause significant bleeding in some patients and even death in a handful. When such bleeding occurs, that does not necessarily mean there was a medical “error.” It might, but even in this case, I can’t help but point out that most injuries to the liver by a percutaneous needle heal themselves. Indeed, ultrasound- or CT-guided liver biopsies are performed using much larger needles than any needle used for a pericardiocentesis, and bleeding is uncommon. (One study pegs it at 0.7%.) It was unfortunate indeed that this patient developed a pseudoaneurysm. Of course, this death might have been due to medical error. Perhaps the physician doing the procedure didn’t take adequate care to avoid lacerating the liver. We just don’t know. Given that the vast majority of bleeding after percutaneous procedures can’t be attributed to medical error, if you define any such bleeding complication as medical error, you’re going to vastly overestimate the true rate of medical error.
So let’s take a look at the some of the most cited studies that make up the data used by Makary and Daniel for their commentary.
The IOM Report: To Err Is Human
This was an issue in the IOM study, To Err Is Human. This study dates way, way back to 1999 and estimated that between 44,000 and 98,000 deaths could be attributed to medical error. The IOM came by its estimate by examining two large studies, one from Colorado and Utah, and the other conducted in New York, that found that adverse events occurred in 2.9% and 3.7% of hospitalizations and that in Colorado and Utah 6.6% of those adverse events led to death, as compared to 13.6% in New York hospitals. In both of these studies, it was estimated that over half of these adverse events resulted from medical errors and therefore could have been prevented. Thus, when extrapolated to 33.6 million admissions to US hospitals in 1997, the results of the IOM study implied that between 44,000 and 98,000 Americans die because of medical errors.
To be honest, I didn’t have that big of a problem with the IOM study. I thought it was a bit too broad in defining what constituted a medical error. Indeed, one of the authors of one of the studies used by the IOM related in a New England Journal of Medicine article in 2000:
In both studies, two investigators subsequently reviewed the data and reclassified the events as preventable or not preventable. Preventability is difficult to determine because it is often influenced by decisions about expenditures. For example, if every patient were tested for drug allergies before being given a prescription for antibiotics, many drug reactions would be prevented. From this perspective, all allergic reactions to antibiotics, which are adverse events according to the studies’ definitions, are preventable. But such preventive testing would not be cost effective, so we did not classify all drug reactions as preventable adverse events.
In both studies, we agreed among ourselves about whether events should be classified as preventable or not preventable, but these decisions do not necessarily reflect the views of the average physician and certainly do not mean that all preventable adverse events were blunders. For instance, surgeons know that postoperative hemorrhage occurs in a certain number of cases, but with proper surgical technique, the rate decreases. Even with the best surgical technique and proper precautions, however, a hemorrhage can occur. We classified most postoperative hemorrhages resulting in the transfer of patients back to the operating room after simple procedures (such as hysterectomy or appendectomy) as preventable, even though in most cases there was no apparent blunder or slip-up by the surgeon. The IOM report refers to these cases as medical errors, which to some observers may seem inappropriate.
Certainly, most surgeons consider this inappropriate. Postoperative hemorrahge is a known complication of any surgery. Many times, when a surgeon takes a patient back for postoperative hemorrhage, no specific cause is found, no obvious blood vessel untied off for example. The very definition of medical errors used in many of these studies will inflate the apparent rate. Physicians know that not every adverse event is preventable or due to medical error. However, choices on how to define medical errors had to be made, and, given the difficulty in determining which adverse events (like postoperative bleeding) are due to physician error, system error, or just the plain bad luck of being the patient for whom an accepted potential complication or adverse event happens, it’s not surprising that a simpler definition of “medical error” is preferred by many investigators.
For all its flaws and the awful “doctors are killing lots of patients” reporting that it provoked, reporting that frustrated many of the investigators who carried out the IOM study because it distracted from the true message of the report, which was to encourage further investigation and a “culture of safety” in hospitals to improve the safety of patient care, the IOM report does deserve a lot of the credit for sparking the movement to improve quality and decrease medical errors over the last 17 years. At the time, I tended to agree with IOM panel member Lucian Leape, MD, who pointed out that, even if the IOM report did greatly overestimate the number of deaths due to medical errors, “Is it somehow better if the number is only 20,000 deaths? No, that’s still horrible, and we need to fix it.” Exactly.
Classen et al: Quadrupling the IOM number
In 2004, another study was published by David Classen et al involving three tertiary care hospitals using, among other measures, the Institute for Healthcare Improvement’s Global Trigger Tool. This study found from four to ten times the number of deaths attributable to medical error than the IOM did; i.e., approximately 400,000 per year. If this were true, then medical errors would be approaching the number two cause of death in the US, cancer, which claims 585,000 people per year.
Classen et al noted that adverse event tracking methods that had frequently been in use at the time of the IOM report missed a lot of adverse events, noting that this tool found up to ten times as many adverse events. This is, of course, not surprising because, regardless of industry or topic, any voluntary reporting system of bad things is going to underreport those bad things. People don’t like admitting that something bad happened; it’s human nature. As a result, after the IOM report, investigators tried to develop automated tools to mine either administrative data (data reported to insurance companies for purposes of reimbursement) for discharge codes that correlate with adverse events:
The Global Trigger Tool uses specific methods for reviewing medical charts. Closed patient charts are reviewed by two or three employees—usually nurses and pharmacists, who are trained to review the charts in a systematic manner by looking at discharge codes, discharge summaries, medications, lab results, operation records, nursing notes, physician progress notes, and other notes or comments to determine whether there is a “trigger” in the chart. A trigger could be a notation indicating, for example, a medication stop order, an abnormal lab result, or use of an antidote medication. Any notation of a trigger leads to further investigation into whether an adverse event occurred and how severe the event was. A physician ultimately has to examine and sign off on this chart review.
Also, Classen et al, like previous investigators, did not really try to distinguish preventable from unpreventable adverse events:
We used the following definition for harm: “unintended physical injury resulting from or contributed to by medical care that requires additional monitoring, treatment, or hospitalization, or that results in death.” Because of prior work with Trigger Tools and the belief that ultimately all adverse events may be preventable, we did not attempt to evaluate the preventability or ameliorability (whether harm could have been reduced if a different approach had been taken) of these adverse events. All events found were classified using an adaptation of the National Coordinating Council for Medication Error Reporting and Prevention’s Index for Categorizing Errors.
There’s the problem right there. Not all adverse events are preventable. We can argue day and night about what percentage of adverse events are potentially preventable; there are sincere disagreements based on evidence on how to determine that number. The problem comes when adverse events are automatically equated with medical errors. The two are not the same. To be fair, Classen et al try not to do this. The authors are very up front that they deem 100% of the adverse events they detected to be potentially preventable. In any case, Classen et al found in 795 hospital admissions in three hospitals and adverse event rate of 33.2% and a lethal adverse event rate of 1.1%, or 9 deaths.
Again, in fairness, I note that Classen et al never extrapolate their numbers to all hospital admissions. Nor did their study differentiate inpatient adverse events or death as due to medical errors (and therefore preventable) or unpreventable. Rather, the purpose of their study was to demonstrate how traditional methods of reporting underestimate adverse events and how the Global Trigger Tool is far more sensitive at detecting such events than voluntary reporting methods. They showed that. None of that stopped Makary and Daniel from taking this one study of less than 1,000 hospital admissions and extrapolating it to 400,000 preventable deaths in hospitals per year. That is the peril from extrapolating from such small numbers.
Landrigan et al: Not as high as Classen, but still too high and not improving
Another study examining the use of the Global Triggering Tool was carried out by Landrigan et al in 10 North Carolina hospitals and published in the NEJM in 2010. I mention it precisely because it uses similar methods to the ones used by Classen et al and comes up with dramatically lower numbers of preventable deaths. It is also a useful study because it examines temporal trends in estimates of harm, asking the question, “Have statewide rates of harm been decreasing over time in North Carolina?” In brief, examining 2,341 hospital admissions over the ten North Carolina hospitals chosen and using internal and external reviewers to judge whether adverse events detected were preventable, Landrigan et al found that there was a reduction in preventable harms identified by external reviewers that did not quite reach statistical significance (P=0.06), with no significant change in the overall rate of harms. This is a depressing finding, although one wonders if the finding might have reached statistical significance if more hospitals had been included.
Overall, despite the lower percentages, the findings of Landrigan et al are not dissimilar to those of Classen et al taking into account that Landrigan et al deemed 63.1% of the adverse events that they identified as preventable, as opposed to the 100% that Classen et al chose. In their 2,341 hospital admissions, Landrigan et al found an adverse event rate of 18.1%, a lethal adverse event rate of 0.6%, and deemed 14 deaths to have been preventable, with the data summarized in a graph below:
I also note that, like Classen et al, Landrigan et al made no effort to extrapolate their findings to the whole of the United States. That was not the purpose of their study. Rather, the purpose of their study was to ask whether rates of adverse events were declining in North Carolina hospitals from 2002 to 2007. None of that stopped Makary and Daniel from extrapolating from Landrigan’s data to close to 135,000 preventable deaths.
HealthGrade’s failing grade
By far the largest study cited by Makary and Daniel is the HealthGrades Quality Study. The HealthGrades study has the advantage of having analyzed patient outcome data for nearly every hospital in the US using data from the Centers for Medicare and Medicaid. Indeed, 37 million Medicare discharges from 2000-2002 were examined using AHRQ’s PSI (Patient Safety Indicator) Version 2.1, Revision 1, March 2004 software application. The authors identified the rates of 16 patient safety incidents relevant to the Medicare population. Four key findings included:
     1.  Approximately 1.14 million total patient safety incidents occurred among the 37 million hospitalizations in the Medicare population from 2000 through 2002.
     2.  The PSIs with the highest incident rates per 1,000 hospitalizations at risk were Failure to Rescue, Decubitus Ulcer, and Post-operative Sepsis. These three patient safety incidents accounted for almost 60% of all patient safety incidents among Medicare patients hospitalized from 2000 through 2002.
     3.  Of the total of 323,993 deaths among patients who experienced one or more PSIs from 2000 through 2002, 263,864, or 81%, of these deaths were potentially attributable to the patient safety incident(s).
     4.  Failure to Rescue (i.e., failure to diagnose and treat in time) and Death in Low Mortality Diagnostic Related Groups (i.e., unexpected death in a low risk hospitalization) accounted for almost 75% of all mortality attributable to patient safety incidents.
One notes that these data were all derived from Medicare recipients, which means that the vast majority of these patients were over 65. Indeed, the authors of the report themselves note that this is so, pointing out that Medicare patients have much higher patient safety incident rates, “particularly for Post-operative Respiratory Failure and Death in Low Mortality DRG where the relative incident rate differences were 85% and 55% higher, respectively, in the Medicare population as compared to all patients.” So just from the fact that this is a study of Medicare recipients, who are much older than non-Medicare recipients, you know that this study is going to skew towards sicker patients and a higher rate of adverse events, even if the care they received was completely free from error. Still, that didn’t stop Makary and Daniels from including this study and estimating 251,000 potentially preventable hospital deaths per year.
Sloppy language, sloppy thinking
No one, least of all I, denies that medical errors and potentially substandard care (again, the two are not the same thing, although there is overlap) are a major problem. If I didn’t believe that, I wouldn’t have devoted so much of my time over the last three years to quality improvement in breast cancer care, and, as I’ve noted before, I’ve frequently been surprised at the variability in utilization of various treatments among hospitals just in my state.
I’ll paraphrase what the IOM said in its 1999 report: You cannot improve what you cannot measure. The problem that we have here is that everybody seems to be using different language and terms about what we are measuring. For example, Makary and Daniels argue:
Human error is inevitable. Although we cannot eliminate human error, we can better measure the problem to design safer systems mitigating its frequency, visibility, and consequences. Strategies to reduce death from medical care should include three steps: making errors more visible when they occur so their effects can be intercepted; having remedies at hand to rescue patients; and making errors less frequent by following principles that take human limitations into account (fig 2⇓). This multitier approach necessitates guidance from reliable data.
It’s hard to disagree with this. Who can argue with the need for reliable data upon which to base our recommendations and efforts to improve quality of care? However, the devil, as they say, is always in the details. Language matters, as well. Adverse events happen even in the absence of medical errors. Many adverse events are not preventable and do not imply medical errors or substandard medical care. Moreover, determining whether a given medical error directly caused or contributed to a given death in the hospital is far from straightforward in most cases. That’s why I don’t like the term “medical errors” in the context of this discussion, except in egregious cases, particularly as it is often used in the lay press, to imply that any potentially preventable death must have been due to an error. Makary and Daniels fall into that trap in perhaps the most quoted part of their BMJ article:
There are several possible strategies to estimate accurate national statistics for death due to medical error. Instead of simply requiring cause of death, death certificates could contain an extra field asking whether a preventable complication stemming from the patient’s medical care contributed to the death. An early experience asking physicians to comment on the potential preventability of inpatient deaths immediately after they occurred resulted in an 89% response rate. Another strategy would be for hospitals to carry out a rapid and efficient independent investigation into deaths to determine the potential contribution of error. A root cause analysis approach would enable local learning while using medicolegal protections to maintain anonymity. Standardized data collection and reporting processes are needed to build up an accurate national picture of the problem. Measuring the consequences of medical care on patient outcomes is an important prerequisite to creating a culture of learning from our mistakes, thereby advancing the science of safety and moving us closer towards the Institute of Medicine’s goal of creating learning health systems.
Note in the first sentence they refer to “death due to medical error,” while in the second sentence they propose asking whether a “preventable complication stemming from the patient’s medical care contributed to the death.” This basically conflates potentially preventable adverse events and deaths with medical errors, when the two are not the same. Rather, I (and many other investigators) prefer to divide such deaths into preventable and unpreventable. Unpreventable deaths include deaths that no intervention could have prevented, such as death from terminal cancer. Preventable deaths include, yes, deaths from medical error, but they also include deaths that, for example, might have been prevented if patient deterioration was picked up sooner. Whether not picking up such deterioration is an “error” or the result of a problem in the system might or might not be clear. Unfortunately, conflating the two, deaths due to medical error and potentially preventable deaths, only provide ammunition to quacks like the one currently engaged in a campaign against me.
How much death is due to medical error, anyway?
I’ll conclude by giving my answer to the question that all of these studies ask, starting with the IOM report: How many deaths in the US are due to medical errors? The answer is: I don’t know! And neither do Makary and Daniels—or anyone else for sure. I do know that there might be a couple of hundred thousand possibly preventable deaths in hospitals, but that number might be much lower or higher depending on how you define “preventable.” I’m also pretty sure that medical errors, in and of themselves, are not the number three cause of deaths. That’s because medical errors rarely occur in isolation from serious medical conditions, which means it’s very to attribute most deaths to primarily a medical error. That number of 250,000 almost certainly includes a lot of deaths that were not primarily due to medical error, given that that’s 9% of all deaths every year.
But it’s even more than that. Whenever you see an estimate of how many deaths are “deaths by medicine,” it’s very helpful to compare that estimate with what we know to assess its plausibility. As I mentioned above, According to the CDC, of the 2.6 million deaths that occur every year in the U.S., 715,000 occur in hospitals, which means that, if Makary’s estimates are correct, 35% of all hospital deaths are due to medical errors. But the plausibility of Makary’s estimate is worse than that. Remember that the upper estimate used by Makary and Daniels is 400,000 inpatient deaths due to medical error. That’s 56%—yes, 56%—of all inpatient deaths? Seriously? It’s just not anywhere near plausible that one-third to over one-half of all inpatient deaths in the US are due to medical error. It just isn’t.
Here are some other things I know. I know that the risk of death and complications is a fairly meaningless number unless weighed against the benefits of medical care, a point that Harriet Hall made long ago, noting for example that an “an insulin reaction counts as an adverse drug reaction, but if the patient weren’t taking insulin he probably wouldn’t be alive to have a reaction.” I also know that Makary’s suggestion that there should be a field on death certificates asking whether a problem or error related to patient care contributed to a patient’s death will be a non-starter in the litigious United States of America, promises of anonymity notwithstanding.
Over the last three years, I’ve learned for myself from firsthand experience just how difficult it is to improve the quality of patient care. I’ve also learned from firsthand experience that nowhere near all adverse outcomes are due to negligence or error on the part of physicians and nurses. None of this is to say that every effort shouldn’t be made to improve patient safety. Absolutely that should be a top health care policy priority. It’s an effort that will require the rigorous application of science-based medicine on top of expenditures to make changes in the health care system, as well as agreement on exactly how to define and measure medical errors. After all, one death due to medical error is too much, and even if the number is “only” 20,000 that is still too high and needs urgent attention to be brought down. Unfortunately, I also know that, human systems being what they are, the rate will never be reduced to zero. That shouldn’t stop us from trying to make that number as close to zero as we can.
ADDENDUM: Here’s a great video that makes similar arguments.

Are medical errors really the third most common cause of death in the U.S.? (2020 edition)
The claim that medical errors are the third leading cause of death in the US has always rested on very shaky evidence; yet it has become common wisdom that is cited as though everyone accepts it. But if estimates of 250,000 to 400,000 deaths due to medical error are way too high, what is the real number? A recently published study suggests that it’s almost certainly a lot lower.
David Gorski on February 3, 2020
Link

I say this at the beginning of nearly every post that I write on this topic, but it bears repeating. It is an unquestioned belief among believers in alternative medicine and even just among many people who do not trust conventional medicine that conventional medicine kills. Not only does exaggerating the number of people who die due to medical complications or errors fit in with the world view of people like Gary Null, Mike Adams, and Joe Mercola, but it’s good for business. After all, if conventional medicine is as dangerous as claimed, then the quackery peddled by the likes of Null, Adams and Mercola starts looking better in comparison. Unfortunately, there are a number of academics more than willing to provide quacks with inflated estimates of deaths due to medical error. The most famous of these is Dr. Martin Makary of Johns Hopkins University, who published a review (not an original study, as those citing his estimates like to claim) estimating that the number of preventable deaths due to medical error is between 250,000 and 400,000 a year, thus cementing the common (and false) trope that “medical error is the third leading cause of death in the US” into the public consciousness and thereby doing untold damage to public confidence in medicine. As I pointed out at the time, if this estimate were correct, it would mean that between 35% and 56% of all in-hospital deaths are due to medical error and that medical error causes between 10% and 15% of all deaths in the US. The innumeracy that is required to believe such estimates beggars the imagination.
Of course, even with academics providing them with hugely inflated estimates of deaths due to medical error, quacks remain unsatisfied. Perhaps the most famous estimate written by quacks is Gary Null’s Death by Medicine, each new version of which increases the estimate of the number of people who die because of medical errors and “conventional medicine,” to the point where his estimate approaches 800,000 deaths per year, or more than one third of all deaths in the US. (I strongly suspect that Null will find a way to get that estimate up over one million before too long.) That’s why it was refreshing to read a new meta-analysis written (PDF) by investigators at Yale University last week. It provides an estimate that’s significantly larger than the last paper on the topic that I discussed, but more than ten-fold lower than the inflated “third leading cause of death” numbers.
Before I discuss the new Yale paper, I will, as I always do, provide a bit of history. The attempt to quantify how many deaths are attributable to medical error began in earnest in 2000 with the Institute of Medicine’s To Err Is Human, which estimated that the death rate due to medical error was 44,000 to 96,000, roughly one to two times the death rate from automobiles. (This is the estimate to which the Yale investigators, led by Craig Gunderson with first author Benjamin Rodwin, compare their estimates.) In response to the study, the quality improvement (QI) revolution began. Every hospital began implementing QI initiatives. Indeed, I was co-director of a statewide QI effort for breast cancer patients for three years. Also, as I mentioned above, the estimates for “death by medicine” seemingly never do anything but keep increasing. They went from 100,000 to 200,000 and now as high as 400,000. Basically, when it comes to these estimates, it seems as though everyone is in a race to see who can blame the most deaths on medical errors, and each time a larger estimate is published the press gobbles it up uncritically. In contrast, each time a study publishes a more reasonable estimate, all we hear are crickets.
How did we get here? As Mark Hoofnagle put it:
Here's the history, the "3rd cause" canard comes from a major frameshift on measuring error, and a questionable algorithmic measurement of error that does not actually detect mistakes but "ripples" in the EMR that are *proxies* for error – ICU admissions, major order changes etc.
— Mark Hoofnagle (@MarkHoofnagle) February 1, 2019
Mark was referring to the use of the Institute for Healthcare Improvement’s Global Trigger Tool, which is arguably way too sensitive. Also, as I explained in my deconstruction of the Johns Hopkins paper, the authors conflated unavoidable complications with medical errors, didn’t consider very well whether the deaths were potentially preventable, and extrapolated from small numbers. Many of these studies also used administrative databases, which are primarily designed for insurance billing and thus not very good for other purposes.
The Yale paper
If the estimates between 200,000 and 400,000 are way too high, what is the real number of deaths that can be attributed to medical error? Here’s where the meta-analysis by Rodwin et al comes in, estimating the number of preventable deaths at just over 22,000 per year. It’s not even in the top ten. In fact, preventable deaths due to medical error represent less than 1% of all deaths. That number is, of course, still too high, and efforts to decrease should and will continue. (It can never be zero, given that medicine is a system run by human beings, who are inherently imperfect and sometimes make mistakes.) However, it’s nowhere near the third leading cause of death in the US.
How did Rodwin et al derive their estimate? First, here’s their rationale:
In 1999, the Institute of Medicine (IOM) published its seminal report on medical errors, To Err Is Human: Building a Safer Health System.1 This widely cited analysis extrapolated from two studies of adverse events in hospitals and concluded that between 44,000 and 98,000 Americans die annually due to preventable medical error. The two referenced studies evaluated deaths from medical error by first determining the frequency of adverse events in hospitals and then separately deciding whether the adverse event was preventable and whether the adverse event caused harm.2, 3 More recently, a report including several additional studies concluded that medical error causes more than 250,000 inpatient deaths per year in the USA, making it the third leading cause of death behind only cancer and heart disease.4
Studies that review series of admissions and determine whether adverse events occurred, whether the events were preventable, and what harms resulted have been criticized for indirectness when used to estimate the number of deaths due to medical error.5, 6 In contrast, studies of inpatient deaths offer a more direct way of estimating the rate of preventable deaths. We undertook a systematic review and meta-analysis of studies that reviewed case series of inpatient deaths and used physician review to determine the proportion of preventable deaths.
To examine the question of how many deaths per year are preventable and possibly due to medical error, the authors carried out a systematic review and meta-analysis and took care to make separate estimates for patients with less than a three month life expectancy and more than a three month life expectancy. (Spoiler alert: They found that the vast majority of preventable deaths occur in patients with less than a three month life expectancy.) They also only included studies in which the included cases were reviewed by physicians to determine if the death was preventable:
All studies of case series of adult patients who died in the hospital and were reviewed by physicians to determine if the death was preventable were included. Non-English studies were included and translated using Google Translate, which has been shown to be a viable tool for the purpose of abstracting data for systematic reviews.10 Studies which evaluated a series of inpatient admissions to determine if there was a preventable adverse event, and then determined if that adverse event contributed to death, such as those included in the 1999 Institute of Medicine report, were excluded. We primarily searched for studies of consecutive or randomly selected inpatient deaths, but also included studies that used cohorts with selection criteria but analyzed these separately. Studies limited to specific populations such as pediatric, trauma, or maternity patients were excluded because our primary research question was to determine the overall rate of preventable mortality in hospitalized patients and these populations are less generalizable.
The winnowing process to select the studies resulted in sixteen studies from a variety of countries that fit the inclusion criteria, eight of which were of random or consecutive groups of patients and eight of which were of cohorts with selection criteria, the latter of which were analyzed separately. Four of the studies examined data from multiple hospitals. Of the eight studies that could be included in a quantitative meta-analysis (the ones analyzing random or consecutive groups of patients), all defined preventable deaths as those that were rated as greater than 50% chance of having been preventable, while seven of the studies used a Likert scale to define preventability while one used a scale of 0–100%. Five studies used multiple reviewers, three of which used consensus to arbitrate differences of opinion, while one used a third reviewer and one used latent class analysis. Six of the studies included adverse events prior to admission.
The results were as follows for the percentages of hospital deaths deemed more likely than not to have been preventable:
The overall pooled rate was 3.1% (95% CI 2.2–4.1%). Individual studies ranged from 1.4 to 4.4% preventable mortality with statistically significant evidence for heterogeneity (I2 = 84%, p  50% likely to have been preventable.23 A study which evaluated 124 patients from the Emergency Department who died within 24 h of admission found that 25.8% of these deaths could have been prevented.29 Another study from 1994 reported that 21.6% of 22 deaths from certain diagnostic groups were at least “somewhat likely” to have been preventable.28 A large recent study from the Netherlands reported 9.4% of 2182 deaths as “potentially preventable.” The remaining studies with selection criteria reported rates of 0.5–6.2% preventable deaths.
And overall:
Overall, our systematic review found eight studies of hospitalized patients that reviewed case series of consecutive or randomly selected inpatient deaths and found that 3.1% of 12,503 deaths were judged to have been preventable. Additionally, two studies reported rates of preventable deaths for patients with at least 3 months life expectancy and reported that between 0.5 and 1.0% of these deaths were preventable. If these rates are multiplied by the number of annual deaths of hospitalized patients in the USA, our estimates equate to approximately 22,165 preventable deaths annually and up to 7,150 preventable deaths among patients with greater than 3 months life expectancy.31
I note that that latter estimate of ~7,000 deaths a year in previously healthy people is pretty close to the estimate of ~5,000 preventable deaths per year noted in a study from last year that I discussed.
So what, specifically, were the errors that led to preventable hospital deaths? I don’t know why the authors buried the table in the supplemental materials, but I dug it out and examined the main causes. (The numbers in parentheses are the ranges of percentages of preventable deaths between the studies examined.) The main causes are:
    • Clinical monitoring or management (6-53%)
    • Diagnostic error (13-47%)
    • Surgery/procedure (4-38%)
    • Drug or fluid-related (4-21%)
    • Other clinical (4-50%)
    • Infection or antibiotic related (2-14%)
    • Supervision (24%, there being only one study citing this as a cause)
    • Technical problem (6-9%)
    • Inpatient fall (6.5%, only one study again)
    • Transition of care (3.2%, only one study again)
Clearly, the range is wide, depending on the hospital and country. The top three don’t surprise me either, although, as I’ve pointed out before, for surgical procedures it’s not always easy to tell if a surgical mistake versus a known complication from the surgery is the cause of death. Even when carried out by expert hands, surgical procedures can cause significant complications (such as bleeding) in some patients and even death in a handful. This is true for even seemingly very low risk procedures. Similarly, diagnostic errors are tricky as well, as the error often only becomes apparent in retrospect. Nonetheless, this analysis does provide an idea of the sorts of medical errors that can result in potentially preventable deaths. Moreover, because the standard was simply that a death was more likely than not to have been due to medical error and thus preventable, the figure of 22K deaths/year is likely an overestimate, given that it includes a lot of deaths whose cause might not have been medical error.
So how do Rodwin et al account for the huge difference between their estimate and the Institute of Medicine’s estimate of 44,000-98,000 preventable deaths due to medical error per year and, in particular, the ludicrously inflated estimates of greater than a quarter of a million deaths that produced the “third leading cause of death”? It’s mainly because they didn’t use trigger tools to look for complications and then make estimates of how likely those complications were to be preventable and to have resulted in the death of the patient:
These results contrast with earlier estimates of medical error which reported higher rates of preventable mortality. The IOM report as well as similar subsequent reviews has reported much higher estimates.4 Numerous authors have criticized these prior estimates for varied methodologic reasons,5, 6 including poorly described methods for determining preventability and causality for death, as well as for indirectness—these studies have in common that they primarily attempt to define the incidence of adverse events in series of hospitalized patients and then secondarily estimate the likelihood that the adverse event was preventable and the likelihood that the adverse event, rather than underlying disease, caused the patient’s death. The studies we reviewed have the advantage of both using as their denominator a series of inpatient deaths rather than admissions and directly assessing the deaths for preventability.
This study is not without limitations, however. For one thing, the studies included rely only on physician judgment to determine whether a given death examined was preventable. Given that there is no agreed-upon standard to determine whether a death was preventable, this methodology introduces potential biases, such as hindsight bias after poor outcomes. This particular bias, sometimes called the “knew-it-all-along” phenomenon, is very common after traumatic events or poor outcomes and describes the tendency of humans, examining an event that’s already happened, to view the outcome as more predictable than it actually was at the time before the outcome occurred, when the people involved were making the decisions that led to the outcome. Also, all determinations were made by retrospective chart review, and anyone who’s ever taken care of patients in a hospital knows that the medical record often lacks important information regarding management and death. Perhaps that’s why the inter-operator reliability between doctors reviewing these charts was consistently in the fair to moderate range in these studies. In any event, hindsight bias would tend to increase the estimate of preventable deaths, as the doctors reviewing the chart, knowing the outcome, might have excessive confidence due to this bias about how predictable the outcome was.
Another factor in this study that tends to inflate the estimates is that 6/8 of the studies included medical errors from prior admissions or outpatient care in their analysis, which could potentially lead to an overestimation of the number of preventable deaths due to care in the hospitalization. Only one study tried to separate out the two, and found that 25% of preventable deaths were related to prior outpatient events. On the other hand, I’d argue that a medical error is a medical error, regardless of when it happened. If a doctor made an error that harmed the patient in the outpatient setting and the patient died in the hospital after being admitted for the harm caused by that error, that’s still a death due to medical error.
There was also an interesting quirk:
A limitation of our study is also the limited geographic representation due to a lack of studies from the USA. The eight studies included in the meta-analysis are from Europe and Canada. The three studies from the USA were not included in the meta-analysis since they used selected cohorts of patients with an oversampling of specific conditions, and thus per protocol were not pooled with studies of consecutive or randomly selected cohorts.
Why do American studies use a selected cohort methodology that oversamples specific conditions, instead of an approach that’s more directly applicable to coming up with good estimates of preventable hospital mortality? Who knows? (Maybe someone out there does.)
Implications of a lower estimate of medical errors
The bottom line is that, if this study is an accurate reflection of the true number of preventable deaths due to medical error (and I think it’s very good), only around 7,150 people who were previously healthy die preventable deaths from medical error, and the vast majority of such deaths occur in people expected not to live more than three months. We’re talking estimates less than an order of magnitude smaller than the “one third of all deaths” trope. This has implications. For instance:
“We still have work to do, but statements like ‘the number of people who die unnecessarily in hospitals is equal to a jumbo jet crash every day’ are clearly exaggerated,” said corresponding author Benjamin Rodwin, assistant professor of internal medicine at Yale.
More importantly, after agreeing that recent high estimates of preventable deaths are not plausible and that only a small fraction of hospital deaths are preventable, undermine the credibility of the patient safety movement, and divert attention from other important patient safety priorities, Rodwin et al write:
Another important implication of our study relates to the use of hospital mortality rates as quality measures. Overall hospital mortality rates and disease-specific mortality rates continue to be reported in many countries in Europe and the USA.32, 33 In the USA, overall hospital mortality rates are reported by the Veterans Health Administration and disease and procedure-specific mortality rates are used by the Centers for Medicare and Medicaid Services (CMS). Disease-specific mortality rates are also used to determine hospital reimbursement as part of CMS’ Hospital Value-Based Purchasing Program. Our results show that the large majority of inpatient deaths are not due to preventable medical error. Given this finding, variation in hospital mortality rates is more likely due to variation in disease severity and non-disease-related factors that affect the location of a patient’s death. Although disease severity is taken into account through the reporting of adjusted mortality rates, numerous critiques have pointed out the limitations of this approach.34,35,36,37
Even if disease patterns and severity were uniform, however, there would likely be variation in hospital mortality rates because of variation in the use of hospitals at the end of life.28, 37 If it is assumed that the vast majority of hospital deaths are unavoidable, then variation in inpatient mortality should be seen as a measure of where patients die, rather than whether they die. Numerous studies have found that many non-disease-related factors affect location of death, including referral to palliative care, home support, living situation, functional status, and patient and family preferences.38
Elsewhere, the authors note that in Norway there is no hospice system and therefore patents are often admitted for end-of-life care, an observation that surprised me. They further pointed out that this could be why the study from Norway that they included in their meta-analysis reported the lowest rate of preventable mortality. Patients admitted for hospice care were considered unpreventable deaths, and this diluted the percentage of preventable deaths, leading to lower percentages of preventable deaths compared to hospitals in countries with hospice systems. In other words—surprise! surprise!—hospital mortality rates are a poor measure of quality for inpatient hospital care.
More importantly, if we’re truly going to improve quality of care and patient safety, it’s important to focus our efforts where they will do the most good. To do that, we need accurate data. Innumerate and highly implausible estimates that result in the “third leading cause of death” trope credulously bandied about by the press and amplified by quacks are actually antithetical to improving quality of care.

Are medical errors really the third most common cause of death in the U.S.? (2019 edition)
The claim that medical errors are the third leading cause of death in the US has always rested on very shaky evidence; yet it’s become common wisdom that is cited as though everyone accepts it. But if estimates of 250,000 to 400,000 deaths due to medical error are way too high, what is the real number? A study published last month suggests that it’s almost certainly a lot lower and has been modestly decreasing since 1990.
David Gorski on February 4, 2019
Link

There is a myth promulgated by both quacks and academics who should know better that medical errors are the third leading cause of death in the United States. You’ll see figures of 250,000 or even 400,000 deaths each year due to medical errors, which would indeed be the third leading cause of death after heart disease (635,000/year) and cancer (598,000/year). When last I discussed this issue three years ago, specifically a rather poor study out of The Johns Hopkins that estimated that 250,000 to 400,000 deaths per year are due to medical errors, I pointed out how these figures are vastly inflated and don’t even make any sense on the surface. For one thing, there are only 2.7 million total deaths per year in the US, which would mean that these estimates, if accurate, would translate into 9% to 15% of all deaths being due to medical errors. Those numbers just don’t make sense. It’s even worse than that, though. This particular study looked at hospital-based deaths, of which there are around 715,000 per year, which would imply that these estimates, if accurate, would mean that medical errors cause between 35% and 56% of all in-hospital deaths, numbers that are highly implausible, something that would be obvious if anyone ever bothered to look at the appropriate denominators. Unfortunately, in the three years since its publication, the Makary study has taken on a life of its own, and it’s basically become commonly accepted knowledge that medical errors are the third leading cause of death, even though this estimate is based on highly flawed studies and these numbers are five- to ten-fold greater than the number of people who die in auto collisions every year.
I see this number popping up in the most unexpected places, mentioned matter-of-factly, as though it were truth that everyone accepts:
Medical errors are NOT the third leading cause of death in the US. For that to be true, one-third to one-half of all hospital deaths would have to be due to medical errors.
Damn, that lie just won't die, and even good reporters fall for it. That's why it's so insidious. https://t.co/XtkP2CX2gY
— David Gorski, MD, PhD (@gorskon) February 1, 2019
Yes, Arthur Allen, a writer I’ve admired since his book Vaccine, casually included that factoid in his story.
The attempt to quantify how many deaths are attributable to medical error began in earnest in 2000 with the Institute of Medicine’s To Err Is Human, which estimated that the death rate due to medical error was 44,000 to 96,000, roughly one to two times the death rate from automobiles. At the time, in response to the study, the quality improvement (QI) revolution began. Every hospital began implementing QI initiatives. Indeed, I was co-director of a statewide QI effort for breast cancer patients for three years. Yet, as Mark Hoofnagle points out in the Twitter thread above, the estimates for “death by medicine” keep increasing. They went from 100,000 to 200,000 and now as high as 400,000. On quack websites, the number is even higher. For instance, über-quack Gary Null teamed with Carolyn Dean, Martin Feldman, Debora Rasio, and Dorothy Smith to write a paper “Death by Medicine,” which estimated that the total number of iatrogenic deaths is nearly 800,000 a year, which would be the number one cause of death, if true and nearly one-third of all deaths in the US. Basically, when it comes to these estimates, it seems as though everyone is in a race to see who can blame the most deaths on medical errors.
How did we get here? As Mark Hoofnagle put it:
Here's the history, the "3rd cause" canard comes from a major frameshift on measuring error, and a questionable algorithmic measurement of error that does not actually detect mistakes but "ripples" in the EMR that are *proxies* for error – ICU admissions, major order changes etc.
— Mark Hoofnagle (@MarkHoofnagle) February 1, 2019
Mark was referring to the use of the Institute for Healthcare Improvement’s Global Trigger Tool, which is arguably way too sensitive. Also, as I explained in my deconstruction of the Johns Hopkins paper, the authors conflated unavoidable complications with medical errors, didn’t consider very well whether the deaths were potentially preventable, and extrapolated from small numbers. Many of these studies also used administrative databases, which are primarily designed for insurance billing and thus not very good for other purposes.
So, if the estimates between 200,000 and 400,000 are way too high, what is the real number of deaths that can be attributed to medical error? How would we go about estimating it? As part of that Twitter exchange, Mark pointed me to a recent publication that suggests how. Not surprisingly, its estimates are many-fold lower than the Hopkins study. Also not surprisingly, it got basically no press coverage. The study was published two weeks ago in JAMA Network Open; it’s by Sunshine et al. out of the University of Washington and is entitled “Association of Adverse Effects of Medical Treatment With Mortality in the United States: A Secondary Analysis of the Global Burden of Diseases, Injuries, and Risk Factors Study“.
The first thing you should note is that the study doesn’t just look at medical errors, but rather all adverse events, and their association with patient mortality. That basically means any adverse event, whether it was due to a medical error or not. The study itself is a cohort study using the Global Burden of Diseases, Injuries, and Risk Factors (GBD) study, which uses the GBD database to estimate changes in the rate of death due to adverse events from 1990 to 2016. This database is described thusly in the paper:
The 2016 GBD study is a multinational collaborative project with an aim of providing regular and consistent estimates of health loss worldwide. Methods for GBD 2016 have been reported in full elsewhere. Briefly, data were obtained from deidentified death records from the National Center for Health Statistics; records included information on sex, age, state of residence at time of death, and underlying cause of death. Causes were classified according to the International Classification of Diseases, Ninth Revision (ICD-9), for deaths prior to 1999 and the International Statistical Classification of Diseases and Related Health Problems, Tenth Revision (ICD-10) for subsequent deaths. Each death was categorized as resulting from a single underlying cause. All ICD codes were mapped to the GBD cause list, which is hierarchically organized, mutually exclusive, and collectively exhaustive.
And:
The GBD methodology also accounts for when ill-defined or implausible causes were coded as the underlying cause of death. Plausible underlying causes of death were assigned to each ill-defined or implausible cause of death according to proportions derived in 1 of 3 ways: (1) published literature or expert opinion, (2) regression models, and (3) initial proportions observed among targets.
If you want more detail about the database, the paper in which it was reported is open access, but here’s a bit about the data sources:
The GBD study combines multiple data types to assemble a comprehensive cause of death database. Sources of data included VR and VA data; cancer registries; surveillance data for maternal mortality, injuries, and child death; census and survey data for maternal mortality and injuries; and police records for interpersonal violence and transport injuries. Since GBD 2015, 24 new VA studies and 169 new country-years of VR data at the national level have been added. Six new surveillance country-years, 106 new census or survey country-years, and 528 new cancer-registry country-years were also added.
Adverse effects of medical treatment (AEMT) were classified into six categories: (1) adverse drug events, (2) surgical and perioperative adverse events, (3) misadventure (events likely to represent medical error, such as accidental laceration or incorrect dosage), (4) adverse events associated with medical management, (5) adverse events associated with medical or surgical devices, and (6) other. The authors used a method known as cause-of-death ensemble modeling (CODEm), a standard analytic tool used in GBD cause-specific mortality analyses. This method was used to generate mortality rate and cause fraction (percentage of all-cause deaths due to a specific GBD cause) estimates for the years 1990 through 2016. Finally, the authors analyzed the cause-of-death chains for all deaths from 1980 to 2014 to determine how frequently AEMT was (1) anywhere within a death certificate’s cause-of-death chain (ie, not underlying cause) and (2) which other contributing causes were most frequently found in the causal chain when AEMT was certified as the underlying cause.
Let’s look at the author’s primary results. First, they found 123,603 deaths (95% UI, 100,856-163,814 deaths) in which AEMT was determined to be the underlying cause of death. I must admit that when I first read that, for some reason I had a brain fart in which I thought the authors were saying that they had found 123,603 deaths per year due to AEMT. (Too much IOM and Hopkins on the brain, I guess.) Actually, that was the total number for the entire period.
Here’s the rest of the primary findings of the study:
The absolute number of deaths in which AEMT was the underlying cause increased from 4180 (95% UI, 3087-4993) in 1990 to 5180 (95% UI, 4469-7436) in 2016. Most of this increase was due to population growth and aging, as demonstrated by a 21.4% decrease (95% UI, 1.3%-32.2%) in the national age-standardized AEMT mortality rate over the same period, from 1.46 (95% UI, 1.09-1.76) deaths per 100 000 population in 1990 to 1.15 (95% UI, 1.00-1.60) deaths per 100 000 population in 2016 (Figure 1A). When not exclusively measured as the underlying cause of death, AEMT appeared in the cause-of-death chain in 2.7% of all deaths from 1980 to 2014, which corresponds to AEMT being a contributing cause for an additional 20 deaths for each death when it is the underlying cause. Mortality associated with AEMT as either an underlying or contributing cause appeared in 2.8% of all deaths.
Let’s unpack this a minute. We’re looking at a number of deaths due to AEMT that’s 50- to nearly 80-fold smaller than the numbers in the Hopkins study. More than that, the number normalized to population is falling, having fallen 21% over 36 years.
So what’s the difference between this study and studies like the Hopkins study and the studies upon which the Hopkins study was based? First, it uses a database designed to estimate the prevalence of different causes of death, rather than for insurance billing. Second, it used rigorous methodology to identify deaths that were primarily due to AEMTs. One thing about this study that makes sense comes from its observation that AEMT is a contributing cause for 20 additional deaths for each death for which it is the underlying cause. For 5,180 deaths in the most recent year, that means 108,780 deaths had an AEMT as a contributing or primary cause that year, which is in line with the IOM estimates. It’s also in line with my assertions that one major issue with previous studies is that the unspoken underlying assumption behind them is that that if a patient had an AEMT during his hospital course it was the AEMT that killed him. As for the studies finding up to 400,000 deaths a year due to medical errors, they are, as Monty Python would say, right out.
Remember, too, that this is a study of all AEMTs, but the authors did try to estimate what proportion of these AEMTs were due to medical error, or, as they put it, “misadventure.” Take a look at this graph, Figure 3 from the paper:
First of all, notice how, not unexpectedly, AEMTs increase with patient age. Older patients, of course, have more medical comorbidities and tend to be more medically fragile, with less room for things to go wrong. Second of all, notice that for all age ranges save one, how small a fraction of the total AEMTs were deemed to have been due to misadventure representing probable medical error. As the authors put it:
In the secondary analysis, in which AEMT was listed as the underlying cause of death, 8.9% were due to adverse drug events, 63.6% to surgical and perioperative adverse events, 8.5% to misadventure, 14% to adverse events associated with medical management, 4.5% to adverse events associated with medical or surgical devices, and 0.5% to other AEMT (eTable 6 in the Supplement). The ranking of the subtypes was stable over time (Figure 3A) but with increasing rates of adverse drug events and decreasing rates of misadventure and surgical and perioperative adverse events. Adverse events related to medical or surgical devices and other AEMT were nearly absent in the 1990s but have been responsible for a stable proportion of overall AEMT since the switch to ICD-10 coding of death certificates. Surgical and perioperative adverse events were the most common subtype of AEMT in almost all age groups and increased in importance with age (Figure 3B); misadventure was the largest subtype in neonates, and adverse drug events predominated in individuals aged 20 to 24 years.
So what we can say from these data are that (1) AEMTs are not uncommon; (2) the vast majority of AEMTs that occur in patients who die aren’t the primary cause of death; (3) only a relatively small fraction of AEMTs are due to misadventure or medical error; and (4) population-adjusted AEMT rates have been slowly decreasing. The study is not bulletproof, of course. No study is. For instance, the GBD approach uses ICD-coded death certificates, which have shown varying degrees of reliability in identifying medical harm. In addition, it is probable that a significant number of deaths involving AEMT are not captured because of incomplete reporting. There are also issues with GBD methodology that might not accurately capture every AEMT:
…the GBD study’s cause classification system that assigns each death to only a single underlying cause means that some events associated with AEMT may be grouped elsewhere. Such groupings are dependent on which ICD code was assigned as the underlying cause. For example, adverse drug events from prescribed opioids leading to death would likely be assigned to the GBD study’s cause of “opioid abuse” (ICD-10 code, F11) or “accidental poisoning” (ICD-10 code, T40) based on the mechanism of death, whereas they are included with medical harm in many other studies based on the association with a prescription. Somewhat analogously, nosocomial infections (ICD-10 code, Y95) are often coassigned with a pathogen or type of infection when responsible for a death, and, because Y95 does not end up as the single underlying cause on such death certificates, they are not classified in the GBD study as AEMT.
So let’s say that this study’s estimates of how many people die from AEMTs and, in particular, from medical misadventure, are better estimates than the “third leading cause of death” studies. (I happen to think that it is, even if it might have somewhat underestimated AEMTs.) Does that mean there’s no problem? Of course not, one death from medical error is too many. Roughly 5,200 deaths a year from AEMT and 108,000 deaths in which an AEMT was contributory are too many. However, we do no one other than quacks any favors by grossly exaggerating the scope of the problem, and several lines of evidence show that deaths due to AEMTs are decreasing modestly, not skyrocketing, as the “death by medicine” crowd would have you believe. We can do better. We should do better. We won’t do better by spreading myths that medical errors are the third leading cause of death.

Anecdotes: Cheaper by the Dozen
David Weinberg on November 21, 2012
Link
A loan officer sets up a meeting with an aspiring entrepreneur to inform him that his application has been denied. “Mr Smith, we have reviewed your application and found a fatal flaw in your business plan. You say that you will be selling your donuts for 60 cents apiece. “Yes” says Mr. Smith, “that is significantly less than any other baker in town. This will give my business a significant competitive advantage!” The loan officer replies, “According to your budget, at peak efficiency the cost of supplies to make each donut is 75 cents, you will lose 15 cents on every donut you sell. A look of relief comes over Mr. Smith’s face as he realizes the loan officer’s misunderstanding. He leans in closer, and whispers to the loan officer “But don’t you see, I’ll make it up in volume.”
If you find this narrative at all amusing, it is likely because Mr. Smith is oblivious to what seems like an obvious flaw in his logic.
A similar error in logic is made by those who rely on anecdote and other intrinsically biased information to understand the natural world. If one anecdote is biased, a collection of 12 or 1000 anecdotes multiplies the bias, and will likely reinforces an errant conclusion. When it comes to bias, you can’t make it up in volume. Volume makes it worse!
Unfortunately human beings are intrinsically vulnerable to bias. In most day to day decisions, like choosing which brand of toothpaste to buy, or which route to drive to work, these biases are of little importance. In making critical decisions, like assessing the effectiveness of a new treatment for cancer, these biases may make the difference between life and death. The scientific method is defined by a system of practices that aim to minimize bias from the assessment of a problem.
Bias, in general, is tendency that prevents unpredjudiced consideration of a question (paraphrased from dictionary.com). Researchers describe sources of bias as systematic errors. A few words about random and systematic errors will make this description clearer.
Random Error
Random errors are unpredictable variations in a measurement. Random errors, by definition are unpredictable in how they may affect any single measurement. For small samples, random errors may lead to an incorrect conclusion. For instance, 4 consecutive “heads” or 4 consecutive “tails” coin flips are not rare, but when they occur, may give the false impression of an unfair coin. Larger samples decrease the likelihood that random errors will result in an errant conclusion. 1000 coin flips will rarely deviate very much from a 50/50 distribution. More data increases the confidence that a sample measurements approximate the true value.
Systematic Errors (a.k.a. bias)
Bias is a non-random (or systematic) error which tends to distort results in a particular direction.
Larger sample sizes are not protective against bias. In fact, larger sample size increases the likelihood that bias will lead to an erroneous conclusion. Lets say that we want to measure the birth weight of a population of inner city newborn babies. Our scale is mis-calibrated in a way that 8 ounces is added to the weight of every child. The will result in a systematic over estimation of the birth weight of the babies. If we weigh lots and lots of babies, we will develop very impressive statistics with very tight standard errors around the mean weight, but they will be wrong because our measurement is biased. If we do not recognize the bias, more data will make us more confident in our data, but will not make our data any less wrong!
Bias is the nemesis of researchers, and is difficult, if not impossible to eliminate completely. The best researchers strive to minimize bias in their study designs, and acknowledge potential sources of bias they cannot eliminate.
Some types of data are more prone to bias than others. The vulnerability to bias is one of the most important qualities in determining the reliability of data derived from different sources of evidence. At the bottom of the hierarchy of reliability are the anecdote and its first cousin, the testimonial.
Anecdotes are narratives of one time events. Because they are singular events, anecdotes of anomalous occurrences are not balanced by averaging effects that results from multiple observations, thus they are highly vulnerable to random errors. So, if we look at enough anecdotes, the errors will neutralize each other, and we can create a true impression of nature, right?……..Wrong! If a collection of anecdotes represented a random sample of independent events, such a strategy might work, but collections of anecdotes are rarely random or independent….quite the opposite is true. Because of the biases intrinsic to anecdotes, a collection is even more likely to reinforce a false conclusion.
The only anecdotes that become part of the collective consciousness are those that are remembered by an observer, deemed worthy of repetition, and then actually communicated by the spoken or written word. There are many factors which influence the recall, interpretation, and reporting of experience. These factors are not random.
One of the principle reasons experiences are recalled and repeated is precisely because they are unexpected or out of the ordinary. Anecdotes of mundane events are unlikely to be repeated. Great significance is often attached rare and unexpected event, and the human mind seems programmed to look for “causes” of these events. Much of the time the events are just rare, but random occurrences (point examples of random error).
Confirmation bias is a tendency to notice and attach significance to information that reinforces ones preconceptions, while dismissing or ignoring contrary information. Confirmation bias has a great influence on which events are remembered and repeated as anecdotes.
Anecdotes are greatly influenced by the image the narrator has of him or herself, and even more so the image he/she wishes to project to the outside world. I call this Bumper Sticker Bias. I regularly see bumper stickers boasting that the driver’s progeny is an honor student a their respective school, but, strangely, I have never seen a sticker proclaiming “My kid was expelled from Central High School”. Bumper stickers are not random expressions of human experiences, and neither are anecdotes.
“Case Reports” are anecdotes enshrined in the medical literature. There is reason to believe that case reports are somewhat more reliable than an anecdotes told by your taxi driver or your cousin (at least when it comes to medical information). Physicians are ostensibly trained observers, but are in no way immune to all the biases ubiquitous to the human condition. For a case report, there is a written document (the medical record) to consult to for re-creation of the narrative. Most journals publish very few if any case-reports and credible journals do so only after peer review. Even published case reports are not given much weight by the medical community, and the articles are infrequently cited in future literature. Case reports can, however, have value. Rabies, for example, is so uniformly fatal, that a single example of an unvaccinated patient surviving infection with rabies is a highly significant case.
There is a familiar skeptical mantra that says “the plural of anecdote is not data.” The original attribution, and even the accuracy of this quote is a little uncertain. I would argue that the plural of anecdote is data; they are just really unscientific, really really unreliable data. Under the right conditions, anecdotes may generate hypotheses. For these hypotheses to be widely accepted, they must be confirmed by more robust data gathering and analytic methods. One of the recurring themes of pseudoscientific medicine, is that hypotheses generated by anecdotes are confirmed by nothing other than an avalanche of more anecdotes. Like the failed entrepreneur introduced at the beginning of this post, the purveyors of these treatments think that augmenting anecdotes with more anecdotes validates their hypotheses. But they are wrong. When it comes to bias, you can’t make it up with volume.

Pseudoscience: The Conspiracy Against Science
An excellent new book examines pseudoscience in 22 essays by prominent scientists from various fields.
Harriet Hall on April 3, 2018
Link
We live in a post-truth, anti-intellectual world where intuition, “common sense,” and fake news are often preferred to scientific evidence and where pseudoscience is often presented as valid science. Assuming that truth exists and is worth searching for, and that science is our most reliable tool in that search, how can we identify pseudoscience and combat it effectively?
Allison B. Kaufman and James C. Kaufman have edited a collection of articles with the title Pseudoscience: The Conspiracy Against Science. The contributors are experts in various fields who have different approaches to the subject. The result is an invaluable volume that examines the cognitive biases that lead to pseudoscience, the history of pseudoscience, the reasons for its wide acceptance, how it is endangering our society, how to recognize it, and how we might reduce its impact.
In the first chapter, David Hecht argues that understanding pseudoscience is as important as debunking it. Science and pseudoscience are opposite ends of one spectrum; we can easily identify the extremes, but there is no clear line separating them. Pseudoscientific beliefs are not as random or indefensible as they seem, and science is not as objective and detached as we would like to think. Science is powerful but imperfect; and until we understand its limitations, we shouldn’t condemn those who choose not to trust it. We must avoid dogmatism and remember that scientific knowledge is always provisional.
In a chapter titled “The Psychology of (Pseudo) Science: Cognitive, Social, and Cultural Factors,” Emilio Lobato and Corinne Zimmerman show that pseudoscientific and scientific beliefs are essentially formed in the same way and are subject to the same cognitive biases and social influences. In a chapter on “The Illusion of Causality: A Cognitive Bias Underlying Pseudoscience,” Fernando Blanco and Helena Matute explain the psychology of causal illusions and argue that they are one of the central problems underlying pseudoscience. Other chapters discuss what pseudoscience costs society, factors that affect scientific soundness, pseudoscience in the mainstream, and science activism.
David Gorski explains how the misguided concept of “integrative” medicine has allowed the infiltration of quackery into academia, taking advantage of the blind spot of evidence-based medicine: biological plausibility. Britt Marie Hermes, a former naturopath turned whistleblower, deconstructs the core principles of naturopathy. Kevin Folta examines the weapons and tactics in the war on crop biotechnology. Chad Orzel argues that scientific failures like cold fusion and the report of faster-than-light neutrinos are a good thing: they demonstrate how science responds to criticism and corrects its mistakes, while pseudoscience fails to engage meaningfully with criticism. Other chapters examine the science and pseudoscience behind the anti-vaccine movement, IQ tests, hypnosis, parapsychology, AIDS denialism, over-protection of children, and the challenges of changing minds. There is an excellent discussion of predatory journals and the failures of peer review. And much more.
Indre Viskontas shows how confirmation bias and pattern recognition affect our search for meaning, and she explains the multiple comparisons issue. In neuroimaging, “By setting a statistical threshold fairly low and not correcting for multiple comparisons, one can find activation in virtually any part of the brain during any task.”
In the final chapter, “Truth Shall Prevail,” Paul Joseph Barnett and James C. Kaufman stress that scientists are searching for the truth, sometimes at great personal cost. “Humility, or a reticence to speak in absolutes, is the single greatest distinction between the sides of pseudoscience and evidence-based science.”
They say, “Responding to pseudoscience takes its toll. Scientists get frustrated when they have to repeat facts, refute the same fallacious arguments, and end up in defensive positions because their detractors are not bound to the same rules. Like nuisance lawsuits, pseudoscience rarely plays to win; the goal is simply not to lose. Getting people to doubt science counts as a victory.”
We clearly have our work cut out for us, but truth will prevail in the end.
In a foreword, Scott Lilienfeld summarizes the valuable lessons in this book:
     1.  We are all subject to cognitive biases.
     2.  We are largely unaware of our biases.
     3.  Science is a systematic set of safeguards against biases.
     4.  Scientific thinking doesn’t come naturally to humans.
     5.  Scientific thinking is exasperatingly domain-specific. Even Nobel Prize winners can fall prey to pseudoscience in fields outside their area of expertise.
     6.  Pseudoscience and science lie on a spectrum.
     7.  Pseudoscience is characterized by a set of fallible, but useful, warning signs such as an absence of self-correction, overuse of ad hoc maneuvers to immunize claims from refutation, use of scientific-sounding but vacuous language, extraordinary claims in the absence of compelling evidence, over-reliance on anecdotal and testimonial assertions, avoidance of peer review, etc.
     8.  Scientific claims can be wrong. Pseudoscientific claims differ from erroneous claims in that they are deceptive: they appear to be scientific, but they are not.
     9.  Scientific and pseudoscientific thinking are cut from the same basic psychological cloth. Heuristics (mental shortcuts or rules of thumb) are invaluable in everyday life, but when misapplied they can lead to mistaken conclusions.
     10.  Skepticism differs from cynicism. Skeptics must guard against dismissing implausible claims out of disconfirmation bias.
He concludes, “These lessons should also remind readers that we are all susceptible to questionable claims, and that science, although hardly a panacea, is ultimately our best bulwark against our own propensities toward irrationality.”
This is a long, dense, heavily referenced book (510 pages). Some of the chapters are written in a more academic, rather dry style and even include (gasp!) equations. Others are more engaging. I especially enjoyed Kevin Folta’s creative and amusing use of language in his chapter on GMOs. Some examples:
    • Plant genetic improvements put the petal to the metal.
    • The unsinkable rubber duckies of pseudoscience.
    • The scientific community yawned.
    • What happens in the petri dish stays in the petri dish
    • I can kill the most aggressive cancer cells in a dish with a hammer. It does not mean hammers cure cancer.
    • A thin film of goo in a dish is not people
    • Pseudoscientific leanings that are more like falling-overs.
Conclusion: lots of good stuff
        ◦ This book is a tour-de-force, a compendium of vital information about science, especially as it pertains to current topics in the media, and about the forces that conspire against science. We all need to know about these things. We ignore them at our peril. The book is an excellent antidote to fake news and a handy reference. It’s not easy going, but it’s well worth the effort.

The Science of Clinical Trials
Steven Novella on August 28, 2013
Link
Science-based medicine is partly an exercise in detailed navel gazing – we are examining the use of science in the practice of medicine. As we use scientific evidence to determine which treatments work, we also have to examine the relationship between science and practice, and the strengths and weaknesses of the current methods for funding, conducting, reviewing, publishing, and implementing scientific research – a meta-scientific examination.
There have been several recent publications that do just that – look at the clinical literature to see how it is working and how it relates to practice.
Dr. Vinay Prasad led a team of researchers through the pages of the New England Journal of Medicine hunting for medical reversals – studies that show that current medical practice is ineffective. Their results were published recently in the Mayo Clinic Proceedings:
Dr. Prasad’s major conclusion concerns the 363 articles that test current medical practice — things doctors are doing today. His group determined that 146 (40.2%) found these practices to be ineffective, or medical reversals. Another 138 (38%) reaffirmed the value of current practice, and 79 (21.8%) were inconclusive — unable to render a firm verdict regarding the practice.
Prasad also found that 27% of published studies looked at existing treatments while 73% studied new treatments.
This does not mean that 40% of current practice or current treatments are useless. There is likely a selection bias in which treatments that are controversial are more likely to be studied than ones that are well established. Also, as David Gorski has already pointed out, this is a study of one journal (the NEJM) and may reflect a publication bias toward high impact articles – showing new treatments that work and reversals of established treatments.
Also, on a positive note, these studies reflect the practice within medicine of studying even treatments that are already in use, and then abandoning those treatments when the evidence shows they don’t work.
The question remains, however – is 40% acceptable? Is this what we would expect from a system that is working well? In an accompanying editorial, John Ioannidis comments:
”Finally, are there incentives and anything else we can do to promote testing of seemingly established practices and identification of more practices that need to be abandoned? Obviously, such an undertaking will require commitment to a rigorous clinical research agenda in a time of restricted budgets,” concludes Dr. Ioannidis. “However, it is clear that carefully designed trials on expensive practices may have a very favorable value of information, and they would be excellent investments toward curtailing the irrational cost of ineffective health care.”
I agree that this highlights the fact that some current practices are useless or less than optimal and need to be re-examined. It also seems prudent to target expensive treatments that may not work.
These results might also suggest that the medical community, in some cases, adopts new treatments prematurely, based on preliminary evidence that is not reliable. Ioannidis himself pointed out that most new treatments do not work and most preliminary evidence on these treatments are false positives. Simmons et al demonstrated how easy it is to inadvertently manufacture positive results by exploiting researcher degrees of freedom. Publication bias is also a known effect pushing the literature toward positive results.
All of this research seems to be pointing in the same direction – we need to compensate for this positive bias in the clinical research and perhaps raise the bar of rigorous evidence before adopting new treatments. Further, we need to retroactively apply this raised bar of evidence to existing treatments which were perhaps adopted prematurely.
Another researcher, Benjamin Djulbegovic, just published another look at the clinical literature in the journal, Nature. He examined 860 phase III efficacy trials and found that in just over half of these trials the new treatments being studied were superior to existing treatments. He concludes:
“Our retrospective review of more than 50 years of randomized trials shows that they remain the ‘indispensable ordeals’ through which biomedical researchers’ responsibility to patients and the public is manifested,” the researchers conclude. “These trials may need tweak and polish, but they’re not broken.”
His primary point is that this ratio is close to ideal. If the new treatments worked the vast majority of the time, then that would indicate that existing evidence is likely sufficient and it would call into question the ethics of doing the study – putting patients on a placebo or less effective treatment. If the efficacy trials demonstrated that the treatment worked in only a small minority of cases, then the possibility of benefit would be unethically too small and that would also call into question the methods we were using to select treatments for large clinical trials.
Roughly a 50-50 split is therefore in the Goldilocks zone for clinical trials. Djulbegovic points out that this ratio allows for the steady incremental advance of medical treatments, even though only about 2-4% of new treatments represent a true breakthrough.
Another recent review of the clinical research, this one by Clinical Evidence, a project of BMJ, concludes that for about half of studied treatments, the efficacy is simply unknown. They point out that this does not mean half of treatments being used, and the data says nothing about the frequency of use of individual treatments – just that systematic reviews of treatments are inconclusive about half the time.
This result is similar to a 2011 review of Cochrane systematic reviews which found that 45% were inconclusive – could not conclude that the treatment either did or did not work.
Conclusion
One simple interpretation of all of this research into clinical trials is that we need to do more research. In the review of Cochrane reviews above, only 2% of reviews concluded that the examined treatment works and no further research is required (although it seems to me that recommending further research is the default conclusion for Cochrane reviews).
We do not, however, have infinite resources with which to conduct clinical research. We will always be making do with insufficient data, and so we must make the most out of the information we have. Part of that is understanding the patterns in the research, and the strengths and weaknesses of various kinds of medical research at every stage.
We now have a very good working knowledge of biases in clinical studies. Generally speaking, they have a huge positive bias. Preliminary studies should therefore be considered unreliable in terms of making clinical decisions, and should only be used as a basis for designing further research.
We need to recognize that only mature rigorous efficacy trials give us any reliable indication of whether or not a treatment works and is superior to doing nothing or to existing treatments. Even these trials, however, are problematic and are subject to bias. We therefore also need to consider plausibility (prior probability).
The threshold for determining that a treatment is well established as beneficial is: scientifically plausible intervention backed by rigorous clinical trials that show a replicable, clinically significant and statistically significant benefit.
In my opinion, the research clearly shows that this is the reasonable standard. This is the primary meaning of science-based medicine.
Meanwhile, there is another movement within mainstream medicine whose primary goal is to move in the exact opposite direction. To soften the standards of scientific evidence, use pragmatic studies as if they were efficacy trials, interpret placebo effects as if they were physiological effects, and to have a “more flexible” concept of medical evidence.
These two opposite trends can exist side-by-side because the latter exists in an “alternative” world, a literal double standard.

Are NIH study sections a waste of time?
Since the nomination of Dr. Jay Bhattacharya for NIH Director, I’ve been seeing a suggestion from certain contrarian doctors for a a “randomized trial” of study sections vs. a “modified lottery” to determine which grant applications are funded by the NIH. Just what the heck is Dr. Vinay Prasad talking about?
David Gorski on December 9, 2024
Link
Regular readers know that I am a longtime critic of Robert F. Kennedy Jr., in particular his promotion of antivaccine propaganda that contributed to real deaths in a poor nation, his promotion of bizarre conspiracy theories about vaccines and COVID-19, his HIV/AIDS denialism, and, well, just about everything about this overprivileged addled fool. That is, of course, why before the election I characterized RFK Jr. as an “extinction-level threat to federal public health programs and science-based health policy.” Nothing has changed my opinion of him since the election. Indeed, I described Kennedy’s nomination for Secretary of Health and Human Services not long after Donald Trump won the 2024 Presidential election as a “catastrophe for public health and medical research.” And so it has been so far, with President-Elect Donald Trump’s picks for high-ranking federal health positions under his guidance having been a collection of antivaxxers, grifters, and quacks. I mean, seriously: Dave Weldon for CDC Director? He was a hardcore antivaxxer—one of the two or three go-to antivaxxers in Congress for the antivaccine movement—back in the days even before RFK Jr. first inflicted the Simpsonwood conspiracy theory on us. And don’t even get me started on Dr. Mehmet Oz for CMS Administrator, a role in which he would oversee the massive Medicare and Medicaid programs, as well as all programs under the Affordable Care Act. (Come to think of it, maybe “antivaxxers, grifters, and quacks” is too polite a characterization of Trump’s picks thus far.)
Then there’s Dr. Jay Bhattacharya, who was recently nominated for NIH Director. Compared to past directors nominated by Presidents of both parties, Bhattacharya is inarguably grossly underqualified, a health economist who, unlike past NIH Directors—seriously, compare him even to the current NIH Director Dr. Monica Bertagnolli—has no record of ever done anything resembling cutting edge biomedical research or run an organization anywhere near as large as the NIH. His sole claim to fame since the pandemic is COVID-19 contrarianism and his attacks on public health as one of the signatories of the Great Barrington Declaration. As you might recall, the GBD was an October 2020 proposal to let the virus rip through the “low risk” younger population, with the futile goal of achieving “natural herd immunity,” with a poorly defined strategy of “focused protection” that would have supposedly kept those most vulnerable to severe disease and death from COVID-19 safe, allowing the presumably young and healthy to die at a much lower rate than the elderly and ill as the virus rampaged through the population. It was a tendentiously libertarian and profoundly social Darwinist approach to “open up the economy” at the expense of disease and death that never would have worked and ultimately did cause enormous damage to public health.
Dr. Jonathan Howard has been doing a bang-up job of calling out certain COVID-19 contrarian physicians who have been of late praising RFK Jr. on their monetized Substacks, to great acclaim from antivaxxers in the comments., all while claiming to be pro-vaccine and blaming provaccine doctors for his ascendency while, risibly, claiming that “sabotaging RFK Jr.’s confirmation will increase vaccine hesitancy.” (Hint: They aren’t, at least not anymore.) Chief among these is, of course, Dr. Vinay Prasad, the UCSF medical oncologist who has made a name for himself by downplaying COVID-19 severity, fear mongering about public health interventions ranging from masking to vaccines (even parroting the antivax message of “do not comply”), and generally weaponizing evidence-based medicine (EBM) against public health science in the name of ideology, even going so far to call for Anthony Fauci and other scientists to be tried and imprisoned (another common antivax refrain). I had thought of piling on a bit more, and this post is (sort of) that, but as I was thinking about what to write today I thought I should zero in on something that Dr. Prasad keeps throwing out there that no other SBM contributor has the experience or background to deconstruct. It’s a small part of the overall promotion of RFK Jr. and his picks from “medical conservatives” like Dr,. Prasad, but it bothers me and, if implemented, would change NIH funding decisions forever—and very likely not in a good way.
Study sections are a waste of time?
I’ve briefly alluded to Dr. Prasad’s idea before, but given Dr. Prasad’s repetition of the idea, more and more I thought I just had to write about it as the issue is, as usual, way more complex than Dr. Prasad portrays it. Is the question of how the NIH doles out research funding to scientists based on their grant applications kind of “niche”? Perhaps, but it’s critically important in a way that even most physicians don’t realize. Prasad began promoting his idea for “reform” of the NIH system for determining who gets research grants on the day after the election, when, among other good things that he thought he saw coming from Trump’s victory, Dr. Prasad wrote on his personal monetized Substack:
The NIH is a failure. It has never tested how to give grant money. We have no idea if the current system is better than modified lottery or other proposal. It has no interest in data transparency, publishing in timely fashion, and reproducibility. The agency also needs a hair cut, and a leader who understands these concerns.
First of all, by any non-ideological measure, the NIH is not a failure. We can argue about whether it’s doing its intended job as well or efficiently as it could, but it’s not a failure. As I put it when I first learned that Dr. Bhattacharya was Trump’s pick for NIH Director, the NIH is the largest public funder of biomedical research in the world and arguably the greatest engine of biomedical research ever created. Whatever its flaws—and, as a human-created government institution, it has many—the NIH is about as close to a scientific meritocracy as one can imagine in government (aside, of course, from centers that were created by political meddling, such as the National Center for Complementary and Integrative Health (NCCIH), which studies the sorts of quackery that RFK Jr. likes). No, that is not to say that the NIH can’t be improved, perhaps improved a lot, but NIH-funded research has, directly or indirectly, contributed to innumerable advances in medicine and basic medical science, including the Human Genome Project, gene editing, advances in the understanding and treatment of cancer, HIV/AIDS, and much more. Indeed, the NIH is the envy of most wealthy industrialized nations.
What makes the NIH a “failure,” then? Dr. Prasad doesn’t say other than that in his opinion the NIH has supposedly “never tested how to give grant money.” If you want a better distillation of what I like to call “EBM fundamentalism,” the belief that randomized controlled trials are the only way to test a hypothesis and anything else is useless, I have a hard time thinking of one. It’s the same belief that COVID-19 contrarians have weaponized against public health in general, and vaccines as well, since the pandemic. Indeed, Dr. Prasad himself invoked “RCTs über alles” when he fell for RFK Jr.’s and Aaron Siri’s incredibly misleading longtime antivax trope about vaccines in the CDC schedule not having been all tested against “saline placebo.” What, though, does Dr. Prasad actually mean by a “modified lottery”? I’ve been trying to figure it out; so I did some searches of both his Substack and the group Substack Sensible Medicine for which he’s one of the main bloggers. Unsurprisingly, when Trump announced his nomination for NIH Director, Dr. Prasad thought that Dr. Jay Bhattacharya was a “superb pick” to run the NIH and brought up this chestnut again:
Jay will run randomized trials to figure out which way to give grants is optimal. One study can be the current system vs. a modified lottery, where anyone who passes a basic test of completeness is entered into a lottery. If 5 years later, there is no difference in publications, citations, h indices or any other metric, then it is safe to say study sections are a waste of time.
This has a little more “meat” to the proposal, but only a little. All that Dr. Prasad has has added are three things:
     1.  Grant applications passing a “test of completeness,” whatever this means.
     2.  Adding metrics to examine to determine if there is any difference between the two groups (publications, h-indices, or “any other metric).
     3.  Adding a timeframe.
Readers out there who have had NIH grants or who have ever evaluated NIH grants on a study section will immediately recognize some serious problems here, the first of which is the timeframe. Five years is too brief. It’s often only in the last couple of years of a five year NIH grant—sometimes just the last year!—that the publications start flowing, mainly because the first three or four years are spend doing the background work that leads to the publications. Second, the choice of metrics is rather odd. H-indices have a lot of problems and can be gamed. Finally, just what the heck is a “test of completeness”? The devil, of course, would be in the details. Does “completeness” just mean that the grant application has all the component parts in it? Would there be any criteria, for instance, about what the background, research plan, and other sections of the grant require? What about the usual requirements by the NIH that the grant applicant demonstrate that the research team assembled (co-investigators, consultants, etc.) can actually do the work proposed and that the institutions included in the grant have the actual facilities and expertise to do what is proposed? How would one assess that. It seems to me that one would need something much like a study section.
Also left out of this discussion is the plethora of grant funding mechanisms that the NIH already has. Reading between the lines, what I think that Dr. Prasad seems to be most concerned with is the “gold standard” flagship NIH grant mechanism, the R01. the grant that all biomedical researchers seek. These grants are awarded for five years for as much as hundreds of thousands of dollars a year for a defined research project and are renewable after each cycle, based on progress reports and a new grant application proposing what the PI will do in the next five year cycle. However, the NIH has a number of other grant mechanisms, including the R21 (for early stage research and pilot projects), trainee grants, and small business grants to encourage the translation of discovery into marketable products.
In fairness, though, just because Dr. Prasad has never actually fleshed out this proposal in any serious way (beyond throwing it out there in listicles in two Substack posts) and has never, to my knowledge, actually written a detailed post about the pros an cons of the current study section system for dispensing grant funding versus other systems, “modified lottery” or some of the other mechanisms proposed (e.g., the Howard Hughes Institute method of funding researchers, not projects, and giving them freedom to do whatever they want with the funding they receive), doesn’t mean that there is no merit to the idea. Still, if even Dr. Prasad can’t seem to marshal evidence and arguments for this idea, you might understand why I seem to view it with all the seriousness of his proposal to test the childhood vaccine schedule with “randomized cluster trials,” another proposal that he never fleshed out, despite challenges to do so. So you see why I suspect that the “modified lottery” proposal for NIH funding is as much shitposting something that sounds good to those without category expertise but is ultimately meaningless.
Still, contrary to the whines of someone like Dr. Adam Cifu, we here at SBM do not just “attack.” We analyze and criticize. It turns out that there exist literature and a debate about this idea. Let’s take a look. One thing I immediately noticed, though, is that almost no one is proposing replacing the current system with a “modified lottery,” just experimenting with one. I did find one article proposing replacing the current system not just with a “modified lottery” in which applications are screed for certain criteria but a “pure lottery,” but I doubt that even Dr. Prasad wants to go that far into fantasyland.
Study section vs. modified lottery
Let’s compare the current method of funding grants with the idea of a “modified lottery.” I’ve discussed in extreme detail the current method before, which involves multiple layers of review and evaluation of the grants by committees called study sections, which gather together scientists, physicians, statisticians, and other relevant experts to evaluate grant applications and then come up with a priority score that is the primary metric used to determine which grants receive funding. (As a matter of disclosure, I have served as an ad hoc member on a number of NIH study sections, most recently this September, as well as on the study sections for private foundations.) Generally, with rare exceptions, the NIH funds as many grants as it can using its budgeted funds, starting with applications that garner the lowest priority score—yes, lower is better in NIH scoring!—and working its way up the list of grant applications by priority score until all its funds budgeted for research grants are allocated. It’s a complex system, with multiple layers to screen and evaluate the grant applications submitted, and then to administer the funding and oversee progress reports and applications for renewal (for grant funding mechanisms that allow renewals). It’s also fairly expensive. The FY 2024 budget for the NIH Center for Scientific Review (CSR), which manages all study sections and grant peer review, was $130 million, although this is a pretty small percentage of the overall NIH budget of $47 billion.
I also note that CSR has long been making changes to how the NIH reviews grants, most notably when Dr. Antonio Scarpa (who was the chairman of the Department of Physiology and Biophysics at Case Western Reserve University when I was there studying for my PhD) ran CSR. Under his leadership, a number of changes were made to the peer review system, some good, some not-so-good, all of which beyond the scope of this post to discuss. My point is that the NIH has been tweaking and trying to improve its peer review system as long as it has had a peer review system. Just because it hasn’t “tested” a system that Dr. Prasad likes does not mean that it is a “failure.” That’s about as unserious a criticism as I can imagine.
In comparison, a “modified lottery” would introduce some randomness into the process by setting up a system that screens grant applications to pick the ones that meet certain predetermined criteria and then randomly choosing from among those applications the grant applications that would receive NIH funding. For example, in a 2016 editorial advocating a modified lottery, Feric Fang and Arturo Casadevall mapped out such a system thusly:
The idea is that “meritorious” equals anything that achieves a certain priority score, say in the best 20% of scores (or, in NIH lingo, lower than the 20th percentile). I’ll let the authors elaborate:
Given overwhelming evidence that the current process of grant selection is neither fair nor efficient, we instead suggest a two-stage system in which (i) meritorious applications are identified by peer review and (ii) funding decisions are made on the basis of a computer-generated lottery (Fig. 1). The size of the meritorious pool could be adjusted according to the payline. For example, if the payline is 10%, then the size of the meritorious pool might be expected to include the top 20 to 30% of applications identified by peer review. This would eliminate or at least alleviate certain negative aspects of the current system, in particular, bias. Critiques would be issued only for grants that are considered nonmeritorious, eliminating the need for face-to-face study section meetings to argue over rankings, which would bring about immediate cost savings. Remote review would allow more reviewers with relevant expertise to participate in the process, and greater numbers of reviewers would improve precision. Funding would be awarded to as many computer-selected meritorious applications as the research budget allows. Applications that are not chosen would become eligible for the next drawing in 4 months, but individual researchers would be permitted to enter only one application per drawing, which would reduce the need to revise currently meritorious applications that are not funded and free scientists to do more research instead of rewriting grant applications. New investigators could compete in a separate lottery with a higher payline to ensure that a specific portion of funding is dedicated to this group or could be given increased representation in the regular lottery to improve their chances of funding. Although the proposed system could bring some cost savings, we emphasize that the primary advantage of a modified lottery would be to make the system fairer by eliminating sources of bias. The proposed system should improve research workforce diversity, as any female or underrepresented minority applicant who submits a meritorious application will have an equal chance of being awarded funding. There would also be benefits for research institutions. A modified lottery would allow research institutions to make more reliable financial forecasts, since the likelihood of future funding could be estimated from the percentage of their investigators whose applications qualify for the lottery. In the current system, administrators must deal with greater uncertainty, as funding decisions can be highly unpredictable. Furthermore, we note that program officers could still use selective pay mechanisms to fund individuals who consistently make the lottery but fail to receive funding or in the unlikely instance that important fields become underfunded due to the vagaries of luck.
To be honest, maybe it’s just a failure in vision, or maybe I’m just not as brilliant as Dr. Prasad (something that Dr. Prasad, given his massive ego, would certainly agree with), but I fail to see how this system, which seems to be something like what Dr. Prasad proposes, improves on the current system, given that determining which grants are “meritorious” under this system would appear to take nearly almost as much effort and resources as the current study section system. In fact, it could quite conceivably require more work. Why? First, as is noted above, they system could included or require more reviewers, who would have to be paid for their time. However, even leaving that aside, under the current system, during study section it is the grants given the higher priority scores by the three or four assigned primary reviewers that are not discussed because they have no chance of being funded. (This is commonly referred to as an application being “triaged.”) The applicants whose grants are triaged receive the written evaluations of the primary reviewers, but, unlike grants considered to be potentially in the fundable range, there is no discussion or debate. In the cases of better-scoring grants that are discussed, the Scientific Review Officer who organizes and runs the study section meeting will write up a detailed summary of the discussion and add it to the reviews by the study section members assigned primary review.
Of course, I’m not even sure that this is the sort of idea that Dr. Prasad means, because he hasn’t bothered to explain what he means beyond buzzwords, other than referring to a “modified lottery.” Prasad’s idea rather reminds me of the Great Barrington Declaration, in which the proposal is so vague that it is impossible to know what he means in practice, although in fairness even the Great Barrington Declaration, as vague and poorly argued as it was, included more detail in its “natural herd immunity strategy with “focused protection” than Dr. Prasad has about his “modified lottery” idea. One can imagine all sorts of variants of the “modified lottery” system that could be proposed, but one thing is for sure. All of them would still require a peer review system resembling the current study section; that is, unless one goes for a “pure lottery,” which not even Dr. Prasad seems to be proposing.
Also, contrary to Dr. Prasad’s have been a number of studies examining whether priority scores for grant applications are predictive of productivity of the PI doing research using the grant funds. Indeed, Feliciani et al recently published a taxonomy of funding mechanisms that involve a lottery, all or in part. Here’s a chart from the review article:
Regarding whatever it is that Dr. Prasad means, I’m quite sure that he doesn’t mean Type 0 or 1 and fairly sure that he doesn’t mean Type 4. What he probably means is Type 2 or 3. Type 3 is a system like the one described in the first article I cited, in which peer review determines which proposals are “meritorious” in the traditional manner, assigning scores to all applications and then choosing a percentage of the best-scored applications based on the anticipated funding line, and then those proposals are funded by lottery. A Type 2 system is a hybrid system in which it is mandated that the funding of proportion of “meritorious” grant applications is determined by peer review, and the rest are placed into a lottery pool. The idea here is to make sure that exceedingly meritorious applications don’t fail to receive funding because of the “bad luck” of a lottery:
Type 2 selection procedures are a different flavor of partial lotteries with focal randomization. While in Type 1 the review panel must come up with a bypass set and may create a lottery pool to resolve ties, in Type 2 the panel must set up a lottery pool.7 The lottery pool typically consists of all eligible proposals deemed to be funding-worthy. Partial lotteries with bypass (Type 2) aim to ensure that excellent proposals are not being missed to bad luck (thanks to the bypass), and also promote a chance for those transformative, innovative or riskier research projects that may be disliked by some reviewers (thanks to the lottery).
Note that Type 2 can be further distinguished by the size of the lottery pool. For example, a funder using a Type 2 procedure may mandate that a number of funding decisions (e.g. 10% of them) is to be determined by lottery (and 90% chosen directly via the bypass): in this case, we can say that the Type 2 lottery is running with a 10% lottery choice rate. A different funder may mandate a 90% lottery choice rate, meaning that 90% of funding awards are chosen from the lottery pool, and only 10% via the bypass. Such lottery choice rates will become important in our simulation experiment and recommendations for funders.
Come to think of it, I don’t think that Dr. Prasad means a Type 2 system. After all, only a fraction (which could range from a small to a large fraction) of “meritorious grants” would be distributed as determined by a lottery.
In any event, the authors actually ran simulations of the various methods. You can read the paper in depth if you want, but, to summarize, there was increasing distributive (“funding should be distributed evenly.”) and unbiased fairness (“funding should be distributed without gender, racial, geographic, or other biases.”) as the investigators modeled methods from Type 0 to type 4 that was associated with decreasing epistemological correctness (“funding should go to the ‘best’ proposals”), the effect being less pronounced in terms of epistemological correctness in the high competition condition (low paylines), which, truth be told, has ben the situation with NIH funding since I’ve been faculty. You can get an idea of what I mean from this figure:
As the authors themselves note, this was a modeling study intended to be used to guide research on these methodologies if they are adopted and thus can provide data. Moreover, balancing fairness versus correctness is a policy decision, not strictly a scientific decision. I will also note that under this model a “pure” lottery would produce the worst epistemological correctness with the most fairness, but is that really what we want in an NIH funding mechanism? If the parameters of this model resemble reality, it is impossible to produce a system that is perfectly fair and perfectly able to pick the “best” grant applications, something that anyone who takes some time to think about the issue (unlike Dr. Prasad) is likely to figure out intuitively.
Indeed, one paper frequently cited to argue that the current NIH system poorly predicts which grants will be the most scientifically productive actually rather makes a point that the NIH is actually pretty good at doing just that, just not at differentiating grants that receive the best priority scores. I’ll cite the passage that shows what I mean:
In contrast, a recent analysis of over 130,000 grant applications funded by the NIH between 1980 and 2008 concluded that better percentile scores consistently correlate with greater productivity (Li and Agha, 2015). Although the limitations of using retrospective publication/citation productivity to validate peer review are acknowledged (Lindner et al., 2015; Lauer and Nakamura, 2015), this large study has been interpreted as vindicating grant peer review (Mervis, 2015; Williams, 2015). However, the relevance of those findings for the current situation is questionable since the analysis included many funded grants with poor percentile scores (>40th percentile) that would not be considered competitive today. Moreover, this study did not examine the important question of whether percentile scores can accurately stratify meritorious applications to identify those most likely to be productive.
We therefore performed a re-analysis of the same dataset to specifically address this question. Our analysis focused on subset of grants in the earlier study (Li and Agha, 2015) that were awarded a percentile score of 20 or better: this subset contained 102,740 grants. This percentile range is most relevant because NIH paylines (that is, the lowest percentile score that is funded) seldom exceed the 20th percentile and have hovered around the 10th percentile for some institutes in recent years.
It is true that the study found that there was little or no difference between grant applications in the best-scored 20% of all applications in terms of the chosen outcome metrics of publications (e.g., publications). In other words, it found that, contrary to what is implied by Dr. Prasad that the NIH study section system is a “failure,” in actuality study sections are quite effective in identifying the most scientifically meritorious applications out of the whole pool of applications. They just can’t stratify the really good applications that garner the best scores finely enough to predict which of them will be most productive; in other words, they’re a (somewhat) blunt tool. Indeed, scientists (including myself) have been saying this sort of thing before, mainly that when paylines are only around the 10th percentile (or, as they have been frequently over the last 20 years, even lower), it becomes a crapshoot—a lottery, if you will—which really excellent grant applications will be funded, and certain really great grant applications will fail to be funded—and not because they aren’t great grant applications that deserve to be funded. After all, what objective criteria are there to distinguish between a grant with a priority score at the 7th percentile versus one at the 8th percentile. Yet, funding decisions need to be made because there is only a finite pool of money, and therefore a cutoff needs to be decided. Indeed, I have frequently said that the solution to this problem, which contributes to NIH study sections being conservative and tending to fund “safer science” is to provide more funding, so that the paylines rise, not necessarily to radically alter the study section system.
Still, even though it’s Dr. Prasad making the proposal, a modified lottery is not necessarily inherently a bad idea, given that it’s far more likely that the Trump administration will cut, rather than increase, the NIH budget, the best case scenario likely being that the NIH sees no significant increase in funding adjusted for inflation during the next four years. The problem, of course, is that it’s hard to figure out just what the heck Dr. Prasad is actually proposing. As far as I can tell based on his advocating that the grants to be distributed by lottery, he seems to be proposing a system more like the Type 4 system in the taxonomy above than a Type 3 system. Remember, the Type 3 system involves the grant pool being winnowed down to the small percentage of grants deemed “meritorious,” perhaps two to three times the number of grants as can be funded, and then that pool being used to pick the winners randomly by lottery. It would involve a lot more than just a “basic test for completeness” to accomplish that.
Then there’s the issue of how one might do a “randomized trial” of the current system versus the system that Dr. Prasad proposes that includes a modified lottery? Does he even have any idea of how difficult such a trial would be to design and carry out? Just think about it a minute. You’d have to set up another parallel system for evaluating grants, and then find a way to randomly assign new grant applications to one or the other. You can bet that a lot of the more prominent and longer-funded NIH grant recipients would do their damnedest to game the system to get their grants assigned to the old system, and you can also bet that the ones who are assigned the new system and fail to win funding will use every last step of the NIH appeal process that the can come up with, not unconvincingly arguing that they were unfairly treated because they weren’t treated like everyone else. Again, just like his risibly bad proposal for a “cluster RCT” of childhood vaccination schedules based on states or counties, Dr. Prasad’s proposal for a randomized trial of the old study section system with his idea (if he can flesh it out) of a “modified lottery reveals just out out of touch with reality he’s become in his ivory tower at UCSF, protected from any hint of critical thinking from the audience that has captured him.
I will conclude this section by reiterating that including some sort of modified lottery in the grant approval process is not an entirely crazy idea. Researchers (myself included) have long complained that, once you get a priority score that’s in the 10th percentile or below, it’s impossible reliably differentiate between them in terms of which applications are more scientifically meritorious. It’s not entirely unreasonable to score the grants, pick the best 20-30% for a lottery, and then decide which ones receive funding based on a lottery or a combination involving picking “the best of the best” in the pool and then using a lottery to choose among the “rest of the best.” Again, though, that does not appear to be what Dr. Prasad is proposing. Truth be told, I don’t think that even Dr. Prasad knows what he’s proposing. He just knows that he really detested the NIH leadership during the pandemic, particularly Anthony Fauci and Frances Collins, and that, just because he didn’t like how the NIH performed during the pandemic (even though the NIH was not responsible for public health interventions or approving vaccines), he concludes that the NIH “has failed” in everything else that it does, particularly peer review of grant applications by its study sections. In this, he has allowed his personal bias added to his EBM fundamentalism, all coupled with his love of shitposting, to lead him to make a profoundly unserious proposal. I like to think that Dr,. Prasad knows better but also knows that his audience does not know better, but I’m starting to wonder if he is genuinely ignorant about how NIH grants are allocated and doesn’t care to educate himself.
Same as it ever was. But why?
An axe to grind?
One wonders why Dr. Prasad would keep repeating this proposal, even though he sure appears unable to say exactly what he means by it. I must confess that I have an idea. To me, it definitely seems that Dr. Prasad has an axe to grind about the NIH based on his having trained there, having done his hematology/oncology fellowship at the National Cancer Institute. Two years ago, when Dr. Prasad was promoted to full professor, he wrote:
Professors should have time to think, but this has been devalued. In a soft money world, I often feel like a Mary Kay salesperson—holding a job with no guaranteed income—where you are perennially hunting for open shifts or grants or teaching slots to fund yourself. I have been deeply fortunate to be grant-funded by an incredibly ambitious and visionary philanthropic organization: Arnold Ventures. At the same time, I have repeatedly seen the limitations of the NIH. That is why I agree with Adam Cifu that with each 5 years after being an associate professor, 10% of your time should be bought out by the university.
This is, of course, a not-uncommon complaint about the current model for biomedical research in academia, although the problem of one’s entire salary only being partially funded by one’s university, with the researcher expected to fund the rest through grants, is primarily an issue for research faculty. The salaries of clinical faculty are usually funded through a combination of university funding plus reimbursements for clinical services rendered to patients, either through an individual “you eat what you kill” model or through a more “socialist” model in which clinical faculty are paid salaries from the pooled clinical income of their departments, with some clinicians who bring in a lot more revenue than their expenses subsidizing the salaries of others—usually those doing a significant amount of research or who are in low-reimbursement specialties (like breast surgery)—who don’t manage to cover their expenses with clinical income. My understanding is that Dr. Prasad is just. 0.2 FTE, meaning that he is only 20% clinical, with 80% being research and everything else; so I get it. Also, tenure means something different to clinical faculty; generally, it only guarantees the university part of their salaries. Unfortunately, most clinical faculty derive most of their salaries from clinical income, which is why I frequently say that, to us, tenure is much less of a big deal, other than the prestige that comes with it.
If he ever were to lose his research funding, Dr. Prasad would likely have to join the many MDs who devote more than 50% of their time to research who ultimately find themselves shifting their activities to more and more clinical work because it’s become more and more difficult to obtain grant funding to cover the academic/research part of their salaries. One also notes that, never having received NIH funding (more on this below), Dr. Prasad seems to have found as an alternative to the NIH an ideologically aligned sugardaddy in Arnold Ventures, a private investment fund created by ex-Enron hedge fund billionaire John Arnold, for projects that help hospitals reduce costs, usually by cutting “unnecessary” or “low-value” care, which aligns pretty well with Dr. Prasad’s opposition to public health interventions. To be fair, though I must point out that, on the other hand, Arnold Ventures has supported Ben Goldacre and that “Second Amendment” activists really hate Arnold Ventures because it also funds research into causes of and solutions to the problem of gun violence; so maybe it’s not all bad. It isn’t the NIH, however, and Arnold Ventures has also supported food crank Gary Taubes, who claims that saturated fat does not contribute to obesity. Back in the day, we wrote a number of articles about Taubes’ claims about sugar and his claim that it isn’t “calories in/calories out” that determines obesity but what you eat, regardless of calories.
In any event, Dr. Prasad continued:
During residency I expected to love cardiology, but it never clicked. Instead, H.G. Munshi, Shuo Ma, and others showed me how fascinating hematology and oncology could be. I went to the NIH with the vague idea that one could combine interest in oncology, EBM and regulatory science, and briefly contemplated working at FDA. But I quickly learned our current Oncology FDA was less committed to EBM than it was to helping for-profit companies bring products to market. Tito Fojo, Susan Bates, Barry Kramer, Sanjeev Bala, Sham Mailankody and many others at NIH influenced my thinking. My virtual mentors grew to the hundreds, as I poured through old issues of JCO, NEJM, JAMA and many others. They included Korn, Freidlin, Tannock, Sargent, Moertel, and countless others. I loved my time at NIH and was very productive there; Cifu and I finished our Reversal book in my final year.
To be brutally honest, I’ve long suspected that one reason why Dr. Prasad is so hostile to the current system of determining which grant applications receive NIH funding comes from his never having received NIH funding as a PI. I went to NIH RePorter, the NIH’s searchable database for all of its funding, where you can look up any grant funded since 1985. A search for “Vinay Prasad” under principal investigator pulled up four active grants, three to Dr. Vinayaka Prasad, a professor in the Department of Microbiology and Immunology at the Albert Einstein College of Medicine, for HIV/AIDS research, the other to the World Health Organization for tobacco regulation research. Expanding the search to all fiscal years in the database brought up lots of grants by Dr. Vinayaka Prasad for HIV/AIDS research at Albert Einstein going back to 1991. Bottom line: It looks as though, unlike myself, Dr. Prasad has never been the PI on an NIH grant. Granted, I haven’t been PI on an NIH grant since 2011; so I understand the frustration.
The difference between Dr. Prasad and me is that I view my inability to be funded by the NIH since 2010 (except as co-investigator on other projects) more as my failure, not a manifestation of the NIH being “broken.” Basically, I was “good enough” scientifically when NIH paylines were a little more generous (as they were in 2005), but no longer “good enough” after they fell to their current low levels. Don’t get me wrong. I’ve discussed problems with NIH funding mechanisms before, mainly its conservatism in terms of funding “safer” science, but that conservatism tends to be a product of low funding. When you can only fund, say, 8% of grant applications that come through, taking a risk becomes more difficult, as I discussed above and here. That I was a casualty of that lower funding is painful, but I nonetheless take the more realistic view that what was once “good enough” no longer is and then try to move on. In contrast, Dr. Prasad blames the system 100% and doesn’t even accept that at least part of the reason why he hasn’t been funded by the NIH is himself.
As I alluded to earlier in this post, when in the course of discussing and deconstructing and refuting a proposal being promoted by an influencer like Dr. Prasad better than the influencer himself can defend (or even describe it), that’s often an indication that the influencer making the proposal is profoundly unserious about it. I believe that’s the case here with Dr. Prasad, just as it was when he said that he thought that RFK Jr. had some good ideas, specifically his opposition to the birth dose of the hepatitis B vaccine and his agreeing with RFK Jr.’s highly deceptive claim and longtime antivax trope implying that the childhood vaccine schedule is not safe because it hasn’t been “adequately tested” against saline placebo, complete with repeated calls for an unethical, impractical, and unfeasible “cluster RCT” of childhood vaccine schedules. Both are claims that betray a lack of knowledge and—dare I say?—seriousness about scientific and policy issues involved.
If that isn’t enough, then consider this. Dr. Prasad and his fellow RFK Jr. sycophants, toadies, and lackeys complain that there is bias in NIH funding. For example, Dr. Prasad decries what he perceives to be the “wokeness” of the NIH in the form of funded studies looking at strategies to provide access to cancer screening tests to homeless people. All that goes along with his rants about “woke” agendas in medical schools. No wonder Dr. Prasad seems utterly unconcerned about proposals being openly proposed by Dr. Bhattacharya that would withhold research grants from schools that Donald Trump and his allies consider too “woke”:
Trump’s nominee to head the National Institutes of Health, Dr. Jay Bhattacharya, a physician and economist at Stanford, reportedly wants to target so called “cancel culture” at a number of top progressive universities, according to The Wall Street Journal.
Those with knowledge of Bhattacharya’s thinking told the newspaper that he’s considering linking the doling out of billions in federal research grants to a measure of “academic freedom” on campuses and punishing those that apparently don’t adequately embrace perspectives championed by conservatives.
Bhattacharya wants to take on what he views as academic conformity in science, which pushed him aside over his criticism of the response to the COVID-19 pandemic, including his opposition to school closures and mask mandates to stop the spread of the virus. He suggested in a Wall Street Journal op ed in 2020 that only up to 40,000 Americans would be killed by the pandemic. More than 1.2 million people died.
Seriously, it’s Lysenko all over again, and these sorts of proposals for ideological tests of purity are a dagger aimed at the heart of biomedical research and utter anathema to the scientific culture at the NIH—to scientific culture in general. Indeed, the only people who have made such proposals before have tended to be ideologically biased legislators (e.g., Sen. Tom Harkin, who. foisted the NCCIH on the NIH, or Rep. Darrell Issa, who tried to torpedo a study of HIV/AIDS because he didn’t like the methodology). That’s yet another reason that NIH study sections are far from a waste of time. They do distinguish scientifically meritorious proposals from those that are lacking; they just aren’t a fine enough measure to reliably distinguish between excellent proposals. That’s why we should continue to study strategies to improve how study sections operate, in order to increase their scientific rigor and decrease any bias, and I’m not against adding the element of a lottery to choosing among proposals that are highly meritorious whose success and output the current system doesn’t predict accurately. However, if some of the anti-“cancel culture” proposals of Dr. Bhattacharya are implemented, such that a schools’ politics matter as much or more than the quality of its research proposals, any improvements or “reforms” made to the NIH peer review system would end up being akin to rearranging the deck chairs on the Titanic after it hits the iceberg of Trumpian political correctness as interpreted by RFK Jr. and Dr. Bhattacharya. Dr. Prasad is unconcerned about that because it’s a form of political correctness that he likes. Most likely, he views it as vindication and just revenge upon his harshest critics.
“Profoundly unserious” doesn’t even begin to describe that, nor does it describe Dr. Prasad’s lack of description of just how one would do a “randomized trial” of the two evaluation systems. Seriously, does Dr. Prasad even think about what he’s proposing? It sure doesn’t look as though he does, but, then, thought is unnecessary, at least scientifically-based thought. He’s not making these proposals for an audience of scientists. He’s making them for an audience of ideologues who don’t know how vaccine schedules are constructed or how the NIH work.

Reporting results from clinical trials is vital for science-based medicine
Clinical trials must report on their outcomes, irrespective their results. Doctors and their patients need all the information, not just the good news stories, to make informed decisions.
Britt Hermes on February 28, 2016
Link

Real medicine is built on high quality data. That’s one lesson I learned through my rehabilitation as a former naturopath. I can assure you that this principle does not apply to naturopathy or any of the other types of alternative medicine, which are mostly based on anachronistic traditions, magical thinking, and poor academic integrity. I now know that real medicine incorporates any therapy scientifically demonstrated to be effective and safe, regardless of origin. Therapies that fail to demonstrate effectiveness remain alternative for good reason.
I made a sharp turn in my career when I realized this, but I was recently surprised to learn that a failure to respect the data can sometimes also characterize medicine. I’m not trying to fool anyone with an appeal to hypocrisy (often used by alternative medicine zealots to argue that their beliefs are somehow valid simply because medicine has problems). But I do want to draw attention to a problem with medical research, which has the potential to undermine science-based medicine and take us down the dark path of misinformed medicine — medicine based on incomplete data.
Most people probably don’t realize that a majority of clinical trials do not report their findings.[1–3] About half of trials conducted before 2007, before it was legally required to register and report clinical trials, have never been published.[4, 5] Even since stronger legal mandates were enacted, most clinical trials still are not registered or reported.[1–3] As a result, there is a huge amount of clinical data unknown to the medical community and not available to clinicians or patients.
This effectively means that medical decisions can be based on incomplete information, not only diminishing the effectiveness of treatments but also eroding the perceived plausibility of medical interventions over time. This is what Sense About Science’s AllTrials campaign is all about: ensuring that all clinical trials register and report results in a publicly-available database. I see it as an obvious and straightforward means to improve medicine.
A brief history of trial registration laws and regulations
The ClinicalTrials.gov registry — a database of clinical trials — was created in 2000 by the National Institutes of Health (NIH) National Library of Medicine with help from the Food and Drug Administration. It was created as a response to the 1997 U.S. Federal law requiring the registration of all clinical trials involving human participants in an open-access database.[6] This was an important legislative step intended to disseminate information about clinical trials to the public and researchers.
Over the past eight years, there have been various policy improvements in reaction to poor trial registration compliance. Most notably, the Food and Drug Administration Amendments Act (FDAAA) Section 801, implemented in 2007, requires sponsors of applicable clinical trials to register and then report results at ClinicalTrials.gov. The trials affected by Section 801 include all non-phase I trials of drugs, medical devices, or biologics that
     1.  began after September 27, 2007 or, for those started earlier, were still underway by December 26, 2007;
     2.  have a research site located in the U.S., or
     3.  run under “an Investigational New Drug application or an Investigational Device exemption.”[2, 3]
The FDAAA stipulated that all trial results must be reported within 12 months of collecting data for the primary research outcome. If a study ends early, any results must be reported within one year of the termination date (some exceptions were made for patent protection).[2] In parallel, the Declaration of Helsinki was amended by the World Medical Association General Assembly to include prospective registration prior to subject recruitment and mandated disclosure of study results to the public.[6]
In January 2015, the National Cancer Institute (NIH) issued a new policy echoing the FDAAA’s requirements to report results within 12 months of the trial’s primary completion date. Importantly, this new policy has been implemented into the conditions for receiving funding.[6]
What is being reported? Not much.
In March 2015, The New England Journal of Medicine published a review by Anderson et al. stating that only 13.4% of clinical trials either completed or terminated between January 1, 2008 and August 31, 2012 reported results within the mandated 12-month time period; only 38.3% of trials had reported results within 60 months (n=13,327).[2] Based on these measly percentages, it is safe to say that many researchers keep their clinical trial results under wraps.
In the midst of rampant underreporting across the board, there is an interesting difference between studies funded by the pharmaceutical industry and those funded by government and academic sources. Based on a random sample of 205 clinical trials out of the 13,327 reviewed by Anderson et al., about 80% of pharmaceutical industry trials reported results or had a legal reason for delay within the 5-year reporting period. In contrast, trials funded by government and academic sources did far worse. Only about 50% of NIH funded trials reported results or had legal reason not to, and other government and academic institutions reported at a rate of about 45%.[2] Overall, the pharmaceutical industry is doing a better job following the law than clinical trials funded by other sources.[1]
It seems that ethical and legal obligations are not drivers for following the rules of reporting.
So what is going on? Despite obligations to report results of clinical trials, researchers clearly fail to do so in a timely manner, if ever. This inaction may be explained by two factors: too much effort and a lack of incentive to publish negative results, and a lack of regulatory oversight.
No carrot: reporting is too much work and there’s no incentive to do it
Reporting results demands effort and when results are negative and unlikely to be published in medical journals, there is little academic incentive to prepare reports for public consumption. Ideally, researchers would budget the time and money needed to report results into their grant proposals, and funding organizations would force researchers to comply, as the NIH is mandated to do, at least on paper.
In a statement to STATnews by a Stanford University communications official, Lisa Lapin, expedient reporting comes down to time and money: Stanford researchers do not have enough time or resources to register and report clinical trial results, which results in 95% of their trial results being reported late or not at all since it became legally required to do so.[1]
This effect seems to spawn from the well-known and well-studied tilt towards positive results known as publication bias, which is in turn often attributed to the cutthroat nature of academia. Scientists are judged on research performance, including number of publications, where their contributions are being published, and the overall impact of their research.[7] In such an environment, positive results seem more interesting and exciting than negative results and are rewarded in what acts as a positive feedback loop.
Outside of academia, clinical trials funded by pharmaceutical companies tend to have more results reported than trials funded by the NIH or other scholarly organizations. This differential makes sense to me given how the pharmaceutical sector is subject to great scrutiny. “Big Pharma” is constantly targeted for alleged misconduct. It is reasonable to think that a pharmaceutical or medical device company would not want to open up vulnerabilities by withholding clinical data. Recently, I happened to meet the high-ranking employee, in charge of this task at an international drug company a few months ago, who reiterated this sentiment. She stated specifically that, amongst her other duties, she keeps track of trial results needing to be reported.
No stick: lack of regulatory oversight and no enforcement of penalties
The other reason for non-industry researchers not reporting data is a lack of regulatory oversight. Penalties for not reporting results have been set by the FDAAA, but they are not enforced. According to Stuart Buck at HealthAffairs.org, the FDAAA law could result in a total collection of penalties over $60 million dollars per day if the law were enforced![8] This figure is based on the $10,000-per-day fine that could be imposed on the 6,599 trials (registered from 2008 to 2012) that failed to report findings on time.
Federal authorities have never enforced the legal requirement to report these results.[8] This lack of enforcement certainly sends the message that the United States does not value the scientific practice of reporting all clinical trial results under the ethical guidelines that scientists would probably claim to respect. The FDA Deputy Commissioner, Dr. Robert Califf, has suggested he will begin to enforce the FDAAA, but so far the FDA has not levied penalties.[1, 9] Curiously, Dr. Califf requested his name be removed from three publications outlining changes for clinical trial management.[10]
The problems with not reporting clinical data
Unpublished research is harmful to medicine and its moral standing. A failure to report trial results is a breach of the ethical obligation to human subjects set forth in the Declaration of Helsinki. Since clinical trial research depends on human volunteers, it is absolutely essential that researchers uphold their end of the deal by making results available, if not for the volunteers, but for others, who may benefit from the findings. Patient advocate AnneMarie Ciccarella reflects this sentiment on the Sense about Science USA website:
We provided our bodies, our tissue samples, our data. I’ve heard the same sentiment expressed many times from patients in clinical trials, ‘This may not help me, but it may help another person.’ It’s time to honor that sentiment.
Undisclosed clinical trial results may not hold answers for better treatment options but may be concealing the ineffectiveness or dangers of treatments. Every patient, physician, and researcher should have access to this vital information that is largely withheld from surveillance. Tragically, an ongoing news story proves to be a powerful example of why no trial should be exempt from registering trial results and basic information, including the identity of the substance under investigation.
On January 17, 2015, a participant of a Phase I clinical trial died after developing brain damage caused by an experimental drug under way in France by the company Biotrial, and five other subjects have been hospitalized for symptoms consistent with brain hemorrhaging. The drug, developed by a Portuguese firm intended to treat a variety of symptoms including anxiety and motor dysfunction associated with Parkinson’s disease as well as chronic pain caused by cancer, was being tested in humans for the first time.[4] As of the writing of this post, neither French authorities nor Biotrial have released specific information regarding the identity of the molecule in the trial.
For now, scientists can only speculate about what substance was being tested in this trial beyond the vague information that the drug is a fatty acid amine hydrolase (FAAH) inhibitor. The truth is that Biotrial has not broken French law or international standards regulating clinical trials.[5] While adverse events in Phase I clinical trials are generally thought to be very rare, we actually do not have data available to independently verify this widely held notion.[5] For now, we can only hope that the French authorities will act in the best interests of these patients and install safeguards against future harm in first-in-human studies.
One obvious step away from problems with trial secrecy is to mandate and enforce trial registration and data sharing of trials at any phase, which includes enough information for the trial to be fully understood pharmacologically and therefore reproducible. The International Committee of Medical Journal Editors (ICMJE) proposed in an editorial published concurrently in 14 major medical journals that authors seeking to publish in these journals must share anonymous individual patient data associated with the results of any clinical trial.[7] The editors state that such patient data is required to reproduce clinical trials and is a tenet of good and ethical scientific practice.
While new policies such as this one are well intentioned, an IPD sharing policy may not support the bigger, and more pressing issue of unavailable trial summary results. Dr. Ben Goldacre, a British physician, science writer, and co-founder of the AllTrials campaign for clinical transparency, summarizes the mismatch between the ICMJE’s policy regarding IPD and scientific need thusly:
Overall, therefore, while greater support for IPD sharing is definitely great news, it’s very odd that all this resource and effort is now going into IPD sharing, when IPD sharing is expensive and not in demand, with complex governance and access control issues, whereas summary results sharing presents no privacy issues, and has been highly in demand, for many decades, but with multiple ongoing barriers to access, and frustratingly slow progress.
The proposals for IPD set forth by the International Committee of Medical Journal Editors are necessary and important. Undoubtedly, policies that uphold ethical obligations in research should be implemented. However, we have a more urgent ethical responsibility. Before venturing into the complex and costly task of sharing IPD, the scientific community needs to address the problem of trial secrecy being perpetuated through lax legal requirements and failure to enforce clinical trial registration, including the drug agent under investigation, followed by timely trial summary results. It seems to be time for an overhaul.
A campaign with solutions
AllTrials is an international campaign demanding responsible and ethical scientific research through the registration and reporting of all past and present clinical trials. Included in its mission is a mandate that it be impossible to receive funding or to sell a product without proving registration of clinical trials first.
Spearheaded by Ben Goldacre, the AllTrials campaign was launched by Sense About Science, BMJ, the Centre for Evidence-Based Medicine, the Cochrane Collaboration, the James Lind Initiative, and PLOS in January 2013 in the UK. In July 2015, the US AllTrials campaign was launched collectively by Sense about Science USA, Dartmouth’s Geisel School of Medicine, and the Dartmouth Institute for Health Policy & Clinical Practice.
The US AllTrials campaign calls for clinical trials to be registered with ClinicalTrials.gov. In total, over 634 medical organizations, including major journals, patient-advocacy groups and professional societies have joined the AllTrials campaign. This is a much needed force for change that if successful would help provide higher transparency for research conducted on human subjects. This is an important moral issue.
If all researchers publicly share negative results in a comprehensive manner (like they are legally and ethically mandated to do) we may start to cultivate a culture of research that embraces humility and honesty. In my opinion, not only will this open modus operandi help researchers resist temptations to cheat, it will also help the public learn to trust the scientific process. Without awareness of what has already been researched, scientists may be wastefully repeating clinical trials or asking questions for which answers could be found in unpublished data. In worst-case scenarios, clinical trials are protected from scrutiny even in the event of participants dying and subsequent investigations impeded by poorly implemented and enforced laws that should be protecting the integrity of patients and promoting the highest ethical standards of scientific research.
Most skeptics would probably agree that one of the most frustrating aspects of discussing science-based medicine with people who do not trust science is trying to convey that, for the most part, science is reliable and that the endeavor is honest. If scientists consistently uphold the ethical duties by reporting all results, the public may be more willing to engage in the scientific enterprise, and researchers will benefit from a greater awareness of existing scientific data to further improve their inquiries and contextualize their findings within a more complete prior corpus of data.
For information about how you can help, visit and AllTrials.net.
References
     1.  Anderson ML, Chiswell K, Peterson ED, Tasneem A, Topping J, Califf RM: Compliance with Results Reporting at ClinicalTrials.gov. N Engl J Med 2015, 372:1031–1039.
     2.  Pansieri C, Pandolfini C, Bonati M: The evolution in registration of clinical trials: a chronicle of the historical calls and current initiatives promoting transparency. Eur J Clin Pharmacol 2015, 71:1159–1164.
     3.  Patients endangered as law is ignored [http://www.statnews.com/2015/12/13/clinical-trials-investigation/]
     4.  Butler D, Callaway E: Scientists in the dark after French clinical trial proves fatal. Nature 2016, 529:263–264.
     5.  French clinical trial debacle highlights weak reporting laws [http://www.statnews.com/2016/01/20/french-clinical-trial-debacle/]
     6.  US Federal Government Efforts to Improve Clinical Trial Transparency with Expanded Trial Registries and Open Data Sharing. AMA J Ethic 2015, 17:1152–1159.
     7.  Taichman DB, Backus J, Baethge C, Bauchner H, de Leeuw PW, Drazen JM, Fletcher J, Frizelle FA, Groves T, Haileamlak A, James A, Laine C, Peiperl L, Pinborg A, Sahni P, Wu S: Sharing Clinical Trial Data — A Proposal from the International Committee of Medical Journal Editors. N Engl J Med 2016:160120050013003.

The fine line between quality improvement and medical research
David Gorski on December 28, 2015
Link

As I’ve mentioned before, the single biggest difference between science-based medicine (SBM) and what I like to call pseudoscience-based medicine, namely the vast majority of what passes for “complementary and alternative medicine” (CAM) or “integrative medicine” is that SBM makes an active effort to improve. It seeks to improve efficacy of care by doing basic and clinical research. Then it seeks to improve the quality of care by applying the results of that research to patient care. Yes, the process is complicated and messy, and it frequently doesn’t progress as fast as we would like it to. Sometimes it goes down blind alleys or takes wrong turns, such as when a treatment is adopted too rapidly and determined later to be ineffective. Overall, however, improvement does occur, and it continues to occur. New treatments that work better are discovered. Old treatments that don’t work as well (or that don’t work at all) are abandoned.
There is, however a blurry line between what constitutes medical research and what constitutes quality improvement (QI). A couple of years ago, in one of those unexpected turns that a career can take, an opportunity presented itself for me to become co-director of a statewide quality improvement consortium for breast cancer care in my state. As I’ve alluded to before, it was a case of unexpectedly being in the right place at the right time, of seeing an opportunity and being willing to take it. How I ended up making quality improvement a large part of my career is unimportant. What is important is that it puts me in a unique position among all the other SBM contributors to discuss the interface between science and quality. (It’s also important that I lay down a disclaimer here that this post represents my opinion and my opinion alone; it does not represent the views of the QI with which I’m affiliated, my cancer center, or my university.) In particular, there are ethical considerations that are not obvious, apparently even to someone as brilliant as Steven Pinker, who Tweeted yesterday:
Once again, “bioethicist” opposes innovation that helps everyone, hurts no one. https://t.co/8SxtgVyBAU
— Steven Pinker (@sapinker) December 27, 2015
I was puzzled by this particular Tweet, which pointed to an article published over the weekend in the New York Times about a new $300 million outpatient surgery center at Memorial Sloan-Kettering Cancer Center, the Josie Robertson Surgery Center, that is touted as being the height of innovation. The article describes a wide variety of practices being tested that range from practices designed to produce a better patient experience, such as tracking badges that allow coordinators to find patients and their families wherever they are, to changes in procedures, such as trying to send patients home earlier, omitting the use of drains, and using those same tracking badges to monitor patient activity and quantify the distances patients walk after their surgery as a means of predicting who can go home sooner or later. It’s a perfect example of how the line between research and quality control can be a bit fuzzy. This appears to be the passage that irritated Pinkerton:
But this race to innovation, bioethicists say, has created a gray area. While federal regulations require researchers to obtain patient consent for participation in clinical trials for novel drugs and devices, hospitals can freely enact internal quality improvement exercises without consent — even if there might be consequences for patient care. Medical centers typically do not inform patients every time they use them to test some new health app, or nursing staff reduction, or data analysis technique — changes that may or may not ultimately benefit the patient’s health. “It is clearly a blurry space,” says Nancy Kass, a bioethics professor who is the deputy director for public health at the Johns Hopkins Berman Institute of Bioethics in Baltimore. “It doesn’t matter if it’s quality improvement or research. The questions we should be asking are: Should we be talking about it? What should we be telling patients about it? What do we know about it that makes us think that it works? What do we know about it to suggest that it is safe, or might be risky, or have some uncertainties?”
My first thought upon reading the passage that provoked Pinker’s criticism was: What does Pinker have against bioethicists? After all, there is nothing in Kass’ statements above that struck me as unreasonable or onerous, which is why I couldn’t for the life of me figure out how Pinker saw this actually rather bland statement of caution that points out how quality improvement efforts can sometimes cross over into a realm where they are not easily distinguishable from human subjects research (HSR) of the sort that requires approval by an institutional review board (IRB). It is a question that all of us involved in quality improvement efforts not infrequently grapple with. Indeed, Pinker’s Tweet reveals a shocking ignorance of some very basic aspects of medicine. In fact, so basic was it that a blog friend of mine pointed out the issue with Pinker’s Tweet:
@sapinker @Wikisteff The thing is: we *don’t know* that it helps everyone and hurts no one. That’s why it’s being studied.
— MarkCC (@MarkCC) December 27, 2015
And:
.@sapinker @Wikisteff and if it’s a study, the participants have a right to know and consent. Why do you see that as a problem?
— MarkCC (@MarkCC) December 27, 2015
Why, indeed? Pinker seems to view annoying ethical considerations about what does and doesn’t constitute medical research and requiring the need for obtaining informed consent as being in “opposition” to “innovation” (whatever “innovation” means in this context, given that everyone and his grandmother claims what they’re doing is “innovation”). It’s also true that there has been and always will be a tension between testing new treatments and procedures and protecting patients from harm, but to look at all “innovation” of the sort being tested in the “ideas laboratory” of MSKCC’s outpatient surgical center as somehow inherently good, helping everyone and hurting no one, is unbelievably naïve. That’s even leaving aside privacy issues presented by systems such as the patient tracking system currently being tested. So what is the difference between research and quality improvement and what are the issues we frequently run into? Let’s take a look.
What is the difference between research and quality improvement?
“Research.” It’s a simple word. We all think we know what it means. However, it has a very specific meaning when we are discussing human subjects research. For the purpose of federal law, which regulates human subjects research and determines what activities are subject to IRB approval and supervision under the Common Rule. Given that most human subjects research in medicine falls under the purview of the Department of Health and Human Services (HHS), it’s instructive to head over to its website and look under 45 CFR §46.102:
(d) Research means a systematic investigation, including research development, testing and evaluation, designed to develop or contribute to generalizable knowledge. Activities which meet this definition constitute research for purposes of this policy, whether or not they are conducted or supported under a program which is considered research for other purposes. For example, some demonstration and service programs may include research activities.
And a human subject is:
(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains
     1.  Data through intervention or interaction with the individual, or
     2.  Identifiable private information.
This is (or at least was) a fairly reasonable definition of human subjects research, although. It’s designed to develop or produce generalizable knowledge. It’s also, unfortunately, almost by necessity somewhat vague in that it’s not always easy to tell what activities are or are not designed to produce generalizable knowledge. Testing a new drug or surgical procedure in a randomized clinical trial is undeniably research under this definition. But what about looking at outcomes of patients treated with different procedures? That’s probably research but it might not be.
Perhaps it will help to contrast quality improvement with research. The American College of Surgeons published a guide with a reasonable differentiation between the two, noting succinctly (which is why I’m quoting it) that research “is carried out to add to the profession’s understanding of surgical conditions and disease in contrast to altering or comparing established or already validated treatment options.” It also notes that QI “is more often designed to study whether an accepted norm or behavior is being conducted locally and, if not, to modify the actions of the personnel involved so as to approach these norms.” In other words, QI is designed to test whether accepted standards of care are being followed, often at a local, single hospital level, and then to initiate processes to decrease the amount of care that is delivered that is not concordant with evidence-based guidelines:
Although QI and HSR often overlap, there are several fundamental differences between the two. First, as already noted, HSR involves generating or contributing to generalizable knowledge, whereas QI attempts to improve a program or service or align current treatment with established best practices and evidence-based medicine. HSR often involves randomization of patients, whereas QI projects typically do not randomize to various treatment arms and more often subject the entire population to a system or policy change, often tracked over time. Whereas HSR generates findings that might affect future policies, QI findings intentionally address current standards or protocols and attempt to change the policies and standards for subsequent patients or encounters. HSR is rooted in identifying a subset of patients to study in the most controlled manner, often using strict inclusion and exclusion criteria to define the population of interest. Conversely, QI is typically all-encompassing for all patients who may have a disease, undergo a procedure, or interact with a specific aspect of the health care system. Exclusion of specific populations or creating exceptions from the algorithm may subvert the QI effort.
Another difference is that human subjects participating in HSR are not guaranteed to benefit. Indeed, they might be harmed, hence the need for truly informed consent that includes a detailed discussion of the risks and potential benefits. In contrast, QI efforts are designed to produce a clear benefit to subsequent patients in terms of safety, efficacy, efficiency, patient satisfaction, quality, or other concretely measurable outcome in the near term. Also, unlike HSR, which usually has a rigid protocol, pre-defined accrual goals, and a defined time frame, QI can be (and should be) continuous. In addition, HSR is basically defined to be published in the scientific literature. QI, on the other hand, frequently remains unpublished and is used internally. That doesn’t mean QI work isn’t published, but usually it’s examples of protocols or measurements of how well a hospital or group of hospitals is adhering to evidence-based guidelines. Finally, QI generally only looks at routine patient care information and does not manipulate patient care, testing one method against another. Here’s a useful decision tree:
Children’s Hospital of Philadelphia also publishes a useful worksheet for differentiating between the two, as do Stanford University and the University of Wisconsin.
So what’s the problem?
The main distinguishing characteristic of QI compared to HSR is its emphasis on standards and testing whether what we as physicians and other health care professionals do when we care for our patients is concordant with currently accepted evidence-based practices and then continuously changing our practices to encourage concordance with science-based medicine guidelines that can be measured with concrete metrics and discourage practices clearly contrary to such guidelines. The devil, as always, is in the details, particularly how adherence to science- and evidence-based guidelines are assessed and how QI corrective measures are implemented in response to findings. When I first signed on to the QI of which I’m now co-director, it was as the clinical champion for my institution, one of 25 others. I represented no more than my institution. Yet, I had to spearhead putting together an IRB application in order for our cancer center to be a part of the QI because, just to be safe, the QI required this of all its member institutions. It’s a requirement that has since been dropped, having been included early on “just to be safe.” In any case, requiring IRB approval for every QI project would definitely inhibit QI activities. On the other hand, it’s definitely a mistake to assume that all QI processes are benign, that all QI processes produce only benefit and do not have the potential to cause harm.
It’s also a mistake to assume that what is going on at MSKCC is only QI. Let’s just put it this way. “Innovation” is not synonymous with quality improvement, contrary to Pinker’s seeming assumption in his reaction to this NYT article. Indeed, almost by definition, “innovation” implies some form of research to test whether that innovation produces better outcomes or better measurable metrics than the old practices did. Certainly, some of what MSKCC is doing doesn’t sound like research. For instance, on its surface the hospital check-in process in and of itself does not sound as though it is HSR in that it’s not really being implemented to improve outcomes or adherence to evidence-based guidelines but rather to make things easier for patients and their families. That, of course, is not the sort of issue that concerns me.
This is:
For instance, the surgery center has done away with some standard medical practices — such as having a designated postoperative recovery unit where specialized nurses monitor patients coming out of anesthesia. Instead, patients will go directly from surgery to private rooms where cross-trained nurses will monitor their recovery from anesthesia.
“The leadership saw this as an opportunity to be a little bit distant from the big-box academic medical center, to test out new work flows, to test out new technology,” Dr. Simon says. “It’s a learning lab for new systems.”
So why is this not a QI project? Because it’s not standard of care. It’s something new, something different from what is normally done. While there are evidence-based guidelines for recovery from anesthesia, there are not agreed-upon evidence-based guidelines regarding such a facility. Moreover, it is not at all unreasonable to be concerned that such a facility could cause harm to some patients. In a private room, unless nurses are assigned to one-on-one coverage (which would be very expensive) it’s more difficult to monitor a patient, and one does have to wonder whether cross-trained nurses will be as adept at recognizing subtle warning signs of trouble in recovering patients that PACU nurses can recognize. Indeed, the leadership even appears implicitly to acknowledge that this is research, testing out “new work flows,” “new technology,” and “new systems.” From what Dr. Simon says, it almost sounds as though the intent of this new outpatient surgery center is to be free from such pesky considerations of whether their tinkering constitutes research and the possibility that someone will demand IRB approval.
Here’s another example:
For instance, Memorial Sloan Kettering administrators have data indicating that elderly patients in the quality improvement program had a higher chance of not being ready to go home the day after their surgeries. At the new center, doctors will study whether additional measures, such as geriatric consults for patients over 75, will improve their chances of a shorter stay.
I’m sorry. This is HSR, plain and simple, by the HHS definition. Clearly, MSKCC is seeking to develop generalizable knowledge about a specific question here. It’s not even really a gray area. Similarly, circling back to those tracking badges, so is this:
For one thing, administrators intend to update the traditional practice of asking patients to walk around soon after surgery. They say they plan to use patients’ locator badges as activity monitors, allowing medical teams to quantify and analyze the distances patients walk. It is a step that may make some patients feel more in control of their recovery — while others may feel more burdened by the added surveillance.
“We don’t know what the data means, because no one has ever measured it before,” Dr. Brett A. Simon, an anesthesiologist who is the director of the surgery center, told me in an interview this month at the new building. Still, he hopes the novel data might eventually be used as a benchmark to help distinguish patients who are recovering on schedule from those who have pain or other symptoms that need to be managed.
“Maybe there’s a predictive value,” Dr. Simon says.
Or maybe, like billions of other data points collected by devices, the distance measurements will prove to be mere noise.
How anyone can view the question of whether the data collected by these tracking badges has predictive value of health outcomes, such as length of stay, as anything other than a research question is beyond me. Dr. Simon is talking about producing generalizable knowledge based on measurements from a new piece of technology; yet this is being justified by this:
So far, about 10,000 patients have gone through the program. But doctors typically do not tell patients that they have been selected for a more streamlined approach to surgical recovery, Dr. Simon says. That is because the actual surgery and medical treatments patients receive have not changed, just related practices.
If you’re going to determine whether a metric measured by new technology can predict whether a patient needs to stay in the hospital longer, that’s not just a “related practice.” That’s part of medical practice. Similarly, whether or not a drain is left in or not is not a “related practice.” It’s part of surgical practice, because the presence or absence of a drain can influence the rate of abscess formation in many surgical procedures.
The most important issue of all
The cardinal rules for any form of HSR include a respect for subject autonomy; clinical equipoise, namely the requirement that there be genuine uncertainty whether the options being tested are better or worse; and procedures at every step of the process to minimize risk to the human research subject. These requirements mandate regulations requiring informed consent, oversight by an ethics board (the IRB), and research designs that minimize the risk of harm to human research subjects. The reason that QI projects are exempt from IRB approval and these other standards is because they involve trying to bring every doctor and hospital up to the science- and evidence-based standard of care, rather than testing new interventions.
What it does not involve is this:
The Josie Robertson center is clearly introducing novel health care approaches that may significantly enhance its patients’ experiences. But given the fact that Memorial Sloan Kettering is a world-renowned institution that many other medical centers tend to follow, it seems remarkable that administrators there have yet to pioneer an equally innovative system for transparently communicating their improvement endeavors to patients.
Dr. Simon says he is working on it.
“We do want to communicate the things we are doing that are new and that improve the patient experience without making them feel alarmed or that they are experimental animals,” Dr. Simon said. “I’m not sure we have 100 percent figured out how to do that.”
This is what I call putting the cart before the horse. This is the sort of thing that should be figured out before implementation, particularly when these sorts of initiatives are coming from a health care institution as revered as MSKCC.
It also doesn’t help that marketing intrudes in a big way in these sorts of programs. As the NYT article clearly shows, much of the reason for the existence of MSKCC’s outpatient surgical center and its high tech initiatives stems not from a medical need, but rather from a need to outcompete its rivals for attracting patients. Given how hypercompetitive an environment medicine in Manhattan has become, this is not surprising. The same thing is happening all over the country, albeit perhaps with not quite the same intensity.
Process improvement is important. Quality improvement in health care is also important. However, protection of patients and any patient who becomes, whether he knows it or not, a human subject in human subjects research trumps both of these. Clearly Steve Pinker does not understand this simple concept. In the meantime, efforts are under way to update the Common Rule for the 21st century. Unfortunately, for purposes of this discussion, the proposed changes deal more with streamlining approvals for minimal risk research and the like than they do with more carefully delineating the difference between QI and HSR. Be that as it may, no one wants to hinder QI efforts unduly, but it has to be remembered that it is real people who can suffer real harm from QI efforts that backfire. This is particularly true, given how much efforts like the outpatient surgery center at MSKCC are driven more by marketing than actual QI or science.
The line between quality improvement and medical research might be a fine one, but it’s not so fine that “innovation” should ever trump protecting patient autonomy and safety. Steve Pinker’s attitude is an all too common one that naïvely views all “innovation” as good for patients and seems oblivious that innovation means little if it isn’t tested in medical research to see if it actually improves patient care.

Screening for disease in people without symptoms: The reality
David Gorski on February 2, 2015
Link
One of the most contentious questions that come up in science-based medicine that we discuss on this blog is the issue of screening asymptomatic individuals for disease. The most common conditions screened for that we, at least, have discussed on this blog are cancers (e.g., mammography for breast cancer, prostate-specific antigen screening for prostate cancer, ultrasound screening for thyroid cancer), but screening goes beyond just cancer. In cancer, screening is a particularly-contentious issue. For example, by simply questioning whether mammography saves as many lives lost to breast cancer as advocates claim, one can find oneself coming under fire from some very powerful advocates of screening who view any questioning of mammography as an attempt to deny “life-saving” screening to women. That’s why I was very interested when I saw a blog post on The Gupta Guide that pointed me to a new systematic review by John Ioannidis and colleagues examining the value of screening as a general phenomenon, entitled “Does screening for disease save lives in asymptomatic adults? Systematic review of meta-analyses and randomized trials.”
Before I get into the study, let’s first review some of the key concepts behind screening asymptomatic individuals for disease. (If you’re familiar with these concepts, you can skip to the next section.) The act of screening for disease is based on a concept that makes intuitive sense to most people, including physicians, but might not be correct for many diseases. That concept is that early intervention is more likely to successfully prevent complications and death than later intervention. This concept is particularly strong in cancer, for obvious reasons. Compare, for example, a stage I breast cancer (less than 2 cm in diameter, no involvement of the lymph nodes under the arm, known as axillary lymph nodes) with a stage III cancer (e.g., a tumor measuring greater than 5 cm and/or having lots of axillary lymph nodes involved). Five year survival is much higher for treated stage I than for treated stage III, and, depending on the molecular characteristics, the stage I cancer might not even require chemotherapy and can be treated with breast conserving surgery (“lumpectomy” or partial mastectomy) far more frequently than the stage III cancer. So it seems intuitively true that it would be better to catch a breast cancer when it’s stage I rather than when it’s stage III.
Unfortunately, that’s not necessarily the case. The reasons are phenomena known as lead time bias, length bias, and overdiagnosis. Lead time bias has been explained multiple times (e.g., here and here), but perhaps the best explanation for a lay public I’ve ever found of lead time bias (although he doesn’t call it that) involves a hypothetical example of cancer of the thumb by Aaron Carroll. Given that cancer survival is measured from the time of diagnosis, if a tumor is diagnosed at an earlier time in its course through the use of a new advanced-screening detection test, the patient’s survival will appear to be longer, even if earlier detection has no real effect on the overall length of survival, as illustrated below:
Lead time bias, in other words, can give the appearance of longer survival even when treatment has no effect whatsoever on the progress of the disease. Patients are simply diagnosed earlier in the disease time course and only appear to live longer when in reality they simply carry the diagnosis screened for longer.
The second concept is length bias. In general, we can’t continually screen for disease so an interval has to be chosen. This introduces a bias. Length bias refers to comparisons that are not adjusted for rate of progression of the disease. The probability of detecting a cancer before it becomes clinically detectable is directly proportional to the length of its preclinical phase, which is inversely proportional to its rate of progression. In other words, slower-progressing tumors have a longer preclinical phase and a better chance of being detected by a screening test before reaching clinical detectability, leading to the disproportionate identification of slowly-progressing tumors by screening with newer, more sensitive tests, with the faster-growing tumors becoming symptomatic “between screenings.” Thus, survival due to the early detection of cancer is a complicated issue.
Finally, there is overdiagnosis. This is a term that refers to disease detected that would likely never progress within the timeframe of the patient’s remaining lifetime to cause a problem. For cancer (for example) it seems completely counterintuitive that there could be overdiagnosis, but there is. As we have learned over the last several years, cancer overdiagnosis is definitely an issue, particularly for the prostate and breast, with perhaps as high as one in three mammography-detected breast cancers in asymptomatic women being overdiagnosed. Basically, this chart illustrates the concept well:
In this chart, which shows growth rates of four hypothetical tumors (A, B, C, and D), tumor D would not be detected because it was growing too fast, while tumor A is growing so slowly that it would likely be overdiagnosed when it reaches the threshold of detectability. Only tumors B and C could potentially benefit from being detected while still confined to the organ. This sort of graph explains why ever-more-sensitive tests that detect disease earlier and earlier have the potential to result in more overdiagnosis.
Finally, overdiagnosis almost inevitably results in overtreatment. Once physicians have detected a disease, be it a cancer, an asymptomatic abdominal aortic aneurysm, or whatever, the onus is on them to treat it.
Now, on to Ioannidis’ review.
Screening, huh! What is it good for? (Probably not absolutely nothing.)
In Ioannidis’ systematic review of meta-analyses and randomized clinical trials, he and coauthors Nazmu and Julianne Saquib take a different approach than I’ve usually seen. Most such systematic reviews and meta-analyses examine screening for only one disease, while this one examines screening for a number of diseases. Also, the key questions to be asked included:
     1.  Does the screening test result in a decrease in mortality due to the disease being screened for (known as disease-specific mortality)?
     2.  Does the screening test result in a decrease in mortality due to all causes?
The background is explained right in the introduction:
Screening for disease is a key component of modern health care. The rationale is simple and attractive—to detect diseases early in asymptomatic individuals and to treat them in order to reduce morbidity, mortality and the associated costs. However, the role of screening often comes into question. Some high-profile controversies have appeared lately in this regard. For example, for breast cancer, the United States Preventive Services Task Force (USPSTF) currently recommends against routine mammographic screening for women aged 40–49 years after retracting its previous recommendation in favour of mammography, as the data failed to show that benefit outweighed harm. The decision against screening drew sharp criticism from various interest groups including patients who overestimate the benefit of screening. Similarly, USPSTF now recommends against screening for prostate cancer in healthy men because harms from prostate specific antigen (PSA) screening exceed the benefit, trials do not show improvement in long-term survival and screening carries a high risk of over-diagnosis with adverse consequences. Again, heated debates have been generated around this change of recommendation, both in the scientific and the popular press.
Some screening tests were entrenched in clinical and public health practice before randomized controlled trials (RCTs) became widely used. As the screening agenda encompasses a large number of tests, and new ones are continuously proposed, it is useful to reassess the evidence supporting their use. Our research question is whether recommended screening tests, among asymptomatic adults, have evidence from RCTs on mortality for diseases where death is a common outcome. In particular, is there evidence of mortality reduction, either disease-specific or all-cause, from screening? To this end, we have compiled and examined systematically the evidence from individual RCTs and meta-analyses thereof for screening tests that have been proposed for detecting major diseases in adults who have no symptoms.
None of this should come as a surprise to regular readers of this blog. But how to answer such a large question regarding the potential benefits and harms of routine screening? Basically, what Saquib et al. did was search the United States Preventive Services Task Force (USPSTF), Cochrane Database of Systematic Reviews, and PubMed, looking for recommendation status, category of evidence, and availability of randomized clinical trials (RCTs) on mortality for screening tests for diseases in asymptomatic adults (excluding pregnant women and children) from the USPSTF. They then identified the relevant RCTs. The chart below summarizes the existing state of evidence identified in the systematic review:
You can examine the data in Tables 1 and 2 yourselves, in which Saquib et al. examined meta-analyses and individual trials for 39 screening tests for 19 diseases in asymptomatic adults. What is very clear is that there is little strong RCT evidence of benefit for disease-specific mortality for many of these modalities. Indeed, The Gupta Guide notes that, for the six diseases/conditions for which the USPSTF recommends screening, only five of them have strong RCT evidence for a reduction in disease-specific mortality: breast, cervical, and colorectal cancer; abdominal aortic aneurysm (AAA); and type 2 diabetes. There were no randomized trials for screening for hypertension.
Basically, in the individual RCTs examined, the risk of disease-specific mortality was reduced in 16 out of 54 tests (30%), while all-cause mortality was reportedly reduced in 4 of 36 RCTs (11%). Examples of tests for which disease-specific mortality was reduced included ultrasound for AAA (42%-73% risk reduction), mammography for breast cancer (0% to 27% risk reduction), and screening for cervical cancer (11% to 48% risk reduction). For the meta-analyses examined, the risk of disease-specific mortality was reduced in analyses of four of eleven tests, but none for all-cause mortality. Examples of screening tests for which disease-specific mortality was reduced included ultrasound for AAA (again, risk reduction of 45%) and mammography (10% to 25% risk reduction). However, none of the meta-analyses showed an all-cause mortality benefit, while all-cause mortality was reduced only by 3%-13% in the RCTs examined. Basically, the findings were disappointing.
The authors note several possible reasons for their findings:
There are many potential underlying reasons for the overall poor performance of screening in reducing mortality: the screening test may lack sufficient sensitivity and specificity to capture the disease early in its process; there are no markedly effective treatment options for the disease; treatments are available but the risk-benefit ratio of the whole screening and treatment process is unfavourable; or competing causes of death do not allow us to see a net benefit. Often, these reasons may coexist. Whether screening saves lives can only be reliably proven with RCTs. However, even for newly proposed tests, we suspect that their adoption in practice may evade RCT testing. A very large number of tests continuously become available due to technological advancement. One may be tempted to claim a survival benefit of screening based on observational cohorts showing improved survival rates, but these are prone to lead-time and other types of bias. Even RCTs can be biased sometimes, as has been discussed and hotly debated in the controversy over mammography.
The authors also note that they did not examine evidence from other trial designs, such as cohort and case-control studies, which could be a potential weakness. Of course, they also make the obvious defense that these studies are generally less robust than RCTs and more prone to biases. The other concession they make is that it can be incredibly difficult to detect reductions in all-cause mortality for the simple reason that the disease being screened for almost always represents only a fraction of causes of death. This means that even a large drop in mortality due to screening for one disease would, even under ideal conditions, result in only a much smaller drop in all-cause mortality. Such a drop is very difficult to detect in an RCT because of the enormous numbers involved.
What is the proper metric to evaluate a screening test?
This point leads us naturally into the discussions of this study. Presented with the study were commentaries by Peter C. Gøtzsche of the Nordic Cochrane Center and Paul Taylor of the Institute of Health Informatics, University College London, entitled “Screening: a seductive paradigm that has generally failed us” and “Tempering expectations of screening: what is the most authoritative advice we can give, given the data that we have?“, respectively. Basically, Gøtzsche, as one might expect based on his previous criticisms of mammography, argues that total mortality should be the primary outcome in screening trials of mortality and that the main focus of screening trials should be to “quantify the harm.” Taylor, on the other hand, argues that in reality the results of Saquib et al. are not so bad, given that 30% of trials showed a disease-specific benefit and even proponents of screening would expect more trials to fail than not. He also points out the difficulties of using all-cause mortality as the primary outcome.
First, let’s see what Gøtzsche argues:
Screening proponents often say that disease-specific mortality is the right outcome, arguing that in order to show an effect on total mortality, trials would become unrealistically large. I believe this argument is invalid, for both scientific and ethical reasons. We do randomized trials in order to avoid bias, and our primary outcome should therefore not be a biased one. Drug interventions are usually more common in a screened group, and they tend to increase mortality for a variety of non-disease related reasons.
While it is true that overtreatment could potentially increase mortality for reasons other than disease, I believe that Gøtzsche is holding screening to an unrealistically high standard. Using his standard, it would be pointless to screen for virtually anything. Why? I like to use breast cancer as an example to illustrate the difficulties of using all-cause mortality as the be-all and end-all for screening. I know the numbers are rough and the analysis simplistic, but the magnitude is illustrative and close enough to give you an idea of the issues involved. This argument takes the form of a simple thought experiment. Consider first that approximately 40,000 women a year die of breast cancer in the US. However, there are approximately 2.5 million total deaths per year, which means that approximately 1.25 million women die every year (estimated to be 1.26 million in 2011). Thus, breast cancer is the cause of approximately 3.2% of female deaths every year. Consequently, if we could prevent 100% of breast cancer deaths (an unrealistic goal), at most we would expect to see a reduction in all-cause mortality of 3.2%.
Aha! I hear some of you saying. You’re counting all female deaths, even those of children, where breast cancer is so incredibly unlikely to be a cause that these deaths should be discounted. Fair enough. Let’s look at women under the age of 40 (the age at which screening begins, making this the group of women for whom screening is instituted with the goal of preventing death from breast cancer). If we subtract the 63,125 deaths that occur per year in women before the age of 40, we’re left with 1,196,875 deaths, which leads us to estimate the number of breast cancer deaths as 3.3% of total deaths of women aged 40 and above—not much different. Of course, some women do die before age 40 of breast cancer. In the US, approximately 1,160 women under 40 die every year of the disease. That brings the proportion of deaths in women over the age of 40 due to breast cancer back down to 3.2%
So let’s say mammography, as an upper estimate, results in a 27% decrease in breast cancer-specific mortality in women over 40. Under ideal circumstances, that would translate into a less than 0.9% decrease in all-cause mortality. The numbers of subjects and years of follow-up in an RCT needed to detect such a small number would be prohibitively expensive. Obviously, for diseases that cause a higher percentage of overall deaths, it will be easier to detect a reduction in all-cause mortality, but for most diseases it’s very difficult indeed to tease out an all-cause mortality except using clinical trials so huge as to be impractical. Of course, this cuts both ways. If you see a decrease in overall mortality due to screening reported that’s bigger than the proportion of deaths expected to be caused by the disease in the age group studied, then something odd is going on, possibly bias.
In any case, I realize my “back of the envelope” calculations are simplistic. They don’t, for example, rigorously consider the trial period and how many deaths would be expected during, for example, one, two, three, or four decades or adjust for age other than in the crudest manner. My point in having done them isn’t to give exact numbers, but rather to illustrate that the question of whether to use disease-specific mortality or all-cause mortality as a primary endpoint in trials screening for diseases that can result in death is not as straightforward a question as Prof. Gøtzsche argues, although he is correct to note that screening is not without harm and that the harms of screening for some diseases can outweigh the benefits. In an ideal world, he’d be correct about all-cause mortality as an endpoint, too, but our world is not ideal, and detecting such small differences in all-cause mortality is often beyond what is feasible and our resources can support.
Taylor, in his response, also notes another confounder:
But there’s the rub. If breast cancer deaths are reduced, but all-cause mortality is unaffected, is this because detecting the latter requires that more statistical power be deployed? Or is it, as Gøtzsche has suggested, because the harms of screening increase deaths from other causes? The most serious cause of harm is overdiagnosis. The independent UK panel took the view that the best estimate of overdiagnosis could be provided by comparing the rates of cancer detection in the screened and the unscreened groups of randomized controlled trials. The problem is that when most trials ended, screening was offered to the women in the control groups, creating overdiagnosis in the follow-up period. The panel therefore restricted their attention to three trials in which no screening was offered to the control group during follow-up. This is a very limited set of data. Saquib, Saquib and Ioannidis ignore the question of harms presumably because there simply are not enough RCT data to review.
In other words, even in these studies, it’s hard to tease out the sorts of data we are interested in so many decades later. Gøtzsche concludes, without presenting concrete evidence to that effect, that the harms of screening negate the benefits, thus resulting in no detectable decline in all-cause mortality. Advocates assume, without proving, that it is a matter of lack of statistical power, as I discussed above, and that bigger trials with more power would detect all-cause mortality decreases. Chances are, it’s both, the relative proportion of each contribution varying according to the specific disease and screening test under study. One thing is certain, though. All screening results in some degree of overdiagnosis. As I’ve said time and time again, whenever you screen for a condition, you will always find a lot more of it. Always. Just consider the 16-fold increase in the incidence of ductal carcinoma in situ since the mammography era began.
There is no doubt in my mind that screening has, in general, been oversold, represented in some cases as a magic bullet that will save far more lives than it actually can. It’s been a relatively recent realization on the part of physicians that overdiagnosis is a real problem because it leads to overtreatment, which can cause harm. Also, in the case of cancer, improvements in treatment could well be blunting any benefits observed due to screening, as Taylor noted in his commentary. On the other hand, that screening has not lived up to its promise does not mean it is useless, as some critics have charged. It is not unreasonable, as Taylor described, to value other outcomes besides mortality. Unfortunately, we’re in the messy and contentious process of trying to determine which screening tests do save lives. As Saquib et al put it:
We argue that for diseases where short- and medium-term mortality are a relatively common outcomes, RCT should be the default evaluation tool and disease-specific and all-cause mortality should be routinely considered as main outcomes. Our overview suggests that even then, all-cause mortality may hardly ever be improved. One may argue that a reduction in disease-specific mortality may sometimes be beneficial even in the absence of a reduction in all-cause mortality. Such an inference would have to consider the relative perception of different types of death by patients (e.g. death by cancer vs death by other cause), and it may entail also some subjectivity. For diseases where mortality outcomes are potentially important but only in the very long term, one has to consider whether the use of other, intermediate outcomes and/or other quasi-experimental designs that may be performed relatively quickly with very large sample sizes (e.g. before and after the introduction of a test) are meaningful alternatives to very long-term RCTs or may add more bias and confusion in a field that has already seen many hot debates. Screening may still be highly effective (and thus justifiable) for a variety of other clinical outcomes, besides mortality. However, our overview suggests that expectations of major benefits in mortality from screening need to be cautiously tempered.
In other words, the science of screening is messy, and we need to be careful not to be too optimistic. Personally, I tend to agree with Taylor, that better risk stratification will be necessary. Screening tends to benefit most populations at high risk for the disease being screened for. Such stratification could allow for—dare I say it?—personalized screening based on individual risk factors.
One wonders how much of that sort of research will be funded in President Obama’s Precision Medicine Initiative, should it be funded. I might have to look into that for a future topic.

Matt Ridley’s not-so-mythical “myth” of basic science
David Gorski on November 1, 2015
Link


I’m a clinician, but I’m actually also a translational scientist. It’s not uncommon for those of us in medicine involved in some combination of basic and clinical research to argue about exactly what that means. The idea is translational science is supposed to be the process of “translating” basic science discoveries in the laboratory into medicine, be it in the form of drugs, treatments, surgical procedures, laboratory tests, diagnostic tests, or anything else that physicians use to diagnose and treat human disease. Trying to straddle the two worlds, to turn discoveries in basic science into usable medicine, is more difficult than it sounds. Many are the examples of promising discoveries that appeared as though they should have led to useful medical treatments or tests, but, for whatever reason, didn’t work when attempted in humans.
Of course, if there’s one thing that the NIH and other funding agencies have been emphasizing, it’s been “translational research,” or, as I like to call it, translation über alles. Here’s the problem. If you don’t have basic science discoveries to translate, then translational science becomes problematic, virtually impossible even. Translational research depends upon a pipeline of basic science discoveries to form the basis for translational scientists to use as the starting point for developing new treatments and tests. Indeed, like many others who appreciate this, I’ve been concerned that in recent years, particularly with tight budgets, the NIH has been overemphasizing translational research at the expense of basic research.
So it was with interest and disappointment that I read Matt Ridley’s latest op-ed in the Wall Street Journal entitled “The Myth of Basic Science.” (Many of you shared my disappointment, to the point where I felt obligated to post a modified and reformatted version of this post—which first appeared on my not-so-super-secret other blog—to SBM.) In his article, Ridley tries to argue that it is innovation and technology that drive scientific discovery, not scientific discovery that drives technological breakthroughs. It’s a profoundly misguided argument that boils down to two central ideas and, more importantly, incorrectly conflates invention and innovation with basic science and failing to understand that innovation in technology often depends on basic science that is decades old. I’m quoting him directly because on Twitter he seems to be claiming that his critics are attacking straw men (they aren’t):
    • “Most technological breakthroughs come from technologists tinkering, not from researchers chasing hypotheses. Heretical as it may sound, ‘basic science’ isn’t nearly as productive of new inventions as we tend to think.”
    • “Governments cannot dictate either discovery or invention; they can only make sure that they don’t hinder it.” (Or, as he quotes others elsewhere, government funding of research is not particularly productive.)
Ridley starts out with what struck me as one of the stranger arguments I’ve ever seen, namely that because there has been so much parallel technological discovery, technology is fast on the way to “developing the kind of autonomy that hitherto characterized biological entities.” (One can’t help but wonder if he’s been watching too many Terminator movies.) Here’s a taste:
Suppose Thomas Edison had died of an electric shock before thinking up the light bulb. Would history have been radically different? Of course not. No fewer than 23 people deserve the credit for inventing some version of the incandescent bulb before Edison, according to a history of the invention written by Robert Friedel, Paul Israel and Bernard Finn.
The same is true of other inventions. Elisha Gray and Alexander Graham Bell filed for a patent on the telephone on the very same day. By the time Google came along in 1996, there were already scores of search engines. As Kevin Kelly documents in his book “What Technology Wants,” we know of six different inventors of the thermometer, three of the hypodermic needle, four of vaccination, five of the electric telegraph, four of photography, five of the steamboat, six of the electric railroad. The history of inventions, writes the historian Alfred Kroeber, is “one endless chain of parallel instances.”
All of which is true. However, the relevance of this observation to basic science being a “myth” is tenuous at best. So where’s the straw man? It comes later in the article:
Politicians believe that innovation can be turned on and off like a tap: You start with pure scientific insights, which then get translated into applied science, which in turn become useful technology. So what you must do, as a patriotic legislator, is to ensure that there is a ready supply of money to scientists on the top floor of their ivory towers, and lo and behold, technology will come clanking out of the pipe at the bottom of the tower.
But, wait, you say. Isn’t that what I just said, that there must be a continual flow of new scientific discoveries to be translated into therapies (or technologies)? Yes, and no. First of all, no one in who knows anything about science believes that “innovation can be turned on and off like a tap” or that you can just throw money at basic scientists and expect technology to come “clanking out of the pipe at the bottom of” the ivory tower. I have no doubt that there are probably politicians who might believe that, but I bet Ridley would be hard pressed to find a scientist involved in applied science, particularly medicine, who believes that. The process is way more complicated. Basic science is hit or miss; you can’t predict what discoveries will or will not be translatable into something useful. In medicine, for instance, it’s virtually impossible to predict whether the discovery of, say, a given enzyme involved in cancer progression will be a useful drug target. Moreover, anyone who knows anything about basic science being translated into useful products knows that both kinds of science are important. You need the basic science as the grist for translational science; there must be a balanced approach. In the case of medicine (and because I’m medical researcher I naturally concentrate mostly on medical research), complaints about the NIH are not that it’s funding translational research but that its emphasis has become unbalanced.
Indeed, unwittingly, Ridley’s examples actually support this view. For the sake of argument, let’s not get into the weeds of whether technological advances are becoming akin to a self-sustaining, evolving system in which human beings a “just along for the ride,” as Ridley puts it, because for what I’m about to say it really doesn’t matter if that’s true or not. (Personally, I think Ridley’s view is exaggerated.) Think about why these various inventions were invented in parallel by so many people, and I bet you’ll see where I’m going with this.
What if the reason for parallel inventions was that the necessary prerequisite discoveries in basic science and engineering had been made, thus making those inventions finally possible? By the early 1800s, the basic physics for photography, for instance, had been around for centuries, dating all the way back to the pinhole camera and the camera obscura. Optics had been worked out for microscopes and telescopes. All that was required was a means of recording images, and that took chemistry, and a number of scientists and inventors were working on that, leading to the Daguerreotype and William Fox Talbot’s silver images on paper. Given that at the time a number of people were working on the problem of photography, it is not surprising that more than one discovered the chemistry that was needed to make photographs a reality.
Elsewhere, Ridley argues:
When you examine the history of innovation, you find, again and again, that scientific breakthroughs are the effect, not the cause, of technological change. It is no accident that astronomy blossomed in the wake of the age of exploration. The steam engine owed almost nothing to the science of thermodynamics, but the science of thermodynamics owed almost everything to the steam engine. The discovery of the structure of DNA depended heavily on X-ray crystallography of biological molecules, a technique developed in the wool industry to try to improve textiles.
This is a profound misunderstanding of how basic science is translated into useful products. For instance, it is true that there were steam engines before the laws of thermodynamics were worked out, and it’s true that the drive to improve steam engine design had a huge influence in the formalization of the laws of thermodynamics in the 19th century. Of course, one has to ask which steam engine Ridley is referring to, given that rudimentary steam engines date back to the first century AD and there were several varieties of steam engines developed in the 17th century. I’m guessing that what he means is the Newcomen engine, developed in 1712. Or perhaps he means James Watt’s steam engine, patented in 1781, which was the precursor to the steam engines that powered ships and industry in the 19th century and beyond. Whichever steam engine he means, Ridley’s description glosses over thermodynamic research done before the steam engine, such as Boyle’s Law, which led to Denis Papin building a steam digester, which was a closed vessel with a tightly fitting lid that built up a high pressure of steam. (In fact, Papin worked closely with Boyle from 1676-1679 to develop the steam digester.) Papin later added a steam release valve that kept his machine from exploding. Watching the valve move up and down, he came up with the idea of a piston and cylinder engine but didn’t follow through with his design himself. That was left to Thomas Savery and later Thomas Newcomen and then, decades later, to James Watt. In the 19th century, the steam engine was an excellent tool that helped scientists formalize the laws of thermodynamics. Basically, discoveries in thermodynamics, such as Boyle’s Law, facilitated designing the steam digester and steam engine and later improving the steam engine. In turn, engineering improvements in the steam engine contributed to the understanding of thermodynamics during the 19th century.
Why did I go through all this? It’s because, even if, as Ridley states, there was a linear view of progress, of translation if you will, from basic science discoveries to products, be they medicines or the steam engine, that view is long gone. It is now understood that basic science drives the development of products and those products drive basic science. So, yes, elucidating the double helical structure of DNA was not possible until the development of X-ray crystallography. So what if X-ray crystallography was originally developed for the wool industry? If I go back another step, X-ray crystallography itself depends on the understanding of so much basic physics that it couldn’t exist until after (1) X-rays were discovered and (2) diffraction patterns and X-ray scattering were understood. These all depended on discoveries taking place over a roughly 25 year period from 1895, when X-rays were discovered, to 1920, by which time the technique of X-ray crystallography had been validated on several crystals. Without the basic science of X-rays, diffraction, scattering, and crystallography, the structure of DNA wouldn’t have been elucidated more than three decades after the 1920s.
All of this leaves aside all the basic science discoveries in genetics and biochemistry that led scientists to know that DNA is the basis of heredity and to know a fair amount about its chemical structure before X-ray crystallography nailed it down. Even taking this view ignores all the science from genetics and biochemistry from the preceding decades that had identified DNA as the basis of heredity, determined its chemical constituents, and gone a long way towards teasing out hints of how DNA might encode information, all information without which the X-ray crystallographic structure would have meant little.
As I said, you never know what basic science will discover, which basic science discoveries will lead to useful products, or ultimately what sorts of uses they will be put to.
Basically, Ridley is attacking a straw man version of basic science, and nothing in his article rebuts the “myth” of basic science, as the title calls it. Through it all, he seems not to understand the difference between R&D, which is “translational research,” research intended to result in a product or the improvement of a product, and research, which is, well, research with the intent of discovering new scientific knowledge. In computer companies, R&D might lead to computer chips. In a car company, R&D might lead to a more efficient engine that can be produced more cheaply. In basic science research, the goal is not nearly as defined, and scientists don’t necessarily know what they will find or where their investigation will take them. In any case, there is nothing contradictory about a bunch of inventors or engineers tinkering and producing inventions like the electric light together, because basic science and technology have to progress enough to produce the prerequisite understanding before such inventions become possible. When that happens, when the conditions are ripe for inventions like the telephone to be invented by several people, it’s because the basic science groundwork has been laid. It might have been laid decades ago and practical applications incrementally developed so that a specific invention becomes possible, or, as in the case with Papin working with Boyle to invent a steam engine, the basic science groundwork and practical application might progress rapidly hand-in-hand.
As I read Ridley’s op-ed, I kept asking myself: What, exactly is he getting at? Why did he choose this example? So what if technological progress happens simultaneously in many places by many people? So what if technology is like a biological species, evolving in response to whatever selective pressures there might be?
Ridley’s purpose becomes clear when he starts citing Terence Kealey, a biochemist turned economist. Now, I had heard of Matt Ridley before, although not recently. He wrote a science book I enjoyed a long time ago, Genome: The Autobiography of a Species in 23 Chapters. However, besides that, I was not very familiar with him. Kealey, on the other hand, I had never heard of. So I Googled him, and I quickly learned that he is an adjunct scholar at the Cato Institute and is known for arguing that government money distorts the scientific enterprise. I also learned that he’s an anthropogenic global climate change denialist and even chaired the Global Warming Policy Foundation, which has been described as the “UK’s most prominent source of climate-change denial” and whose “review” of temperature records has been seriously criticized as incompetent and ideologically-driven.
OK, so Kealey is an anthropogenic climate change denialist, which casts his critical thinking skills with respect to science in great doubt, but maybe he knows economics:
For more than a half century, it has been an article of faith that science would not get funded if government did not do it, and economic growth would not happen if science did not get funded by the taxpayer. It was the economist Robert Solow who demonstrated in 1957 that innovation in technology was the source of most economic growth—at least in societies that were not expanding their territory or growing their populations. It was his colleagues Richard Nelson and Kenneth Arrow who explained in 1959 and 1962, respectively, that government funding of science was necessary, because it is cheaper to copy others than to do original research.
“The problem with the papers of Nelson and Arrow,” writes Mr. Kealey, “was that they were theoretical, and one or two troublesome souls, on peering out of their economists’ aeries, noted that in the real world, there did seem to be some privately funded research happening.” He argues that there is still no empirical demonstration of the need for public funding of research and that the historical record suggests the opposite.
This argument is strange. For one thing, no one that I’m aware of claims that “science would not get funded if the government didn’t do it and economic growth would not happen if science did not get funded by the taxpayer.” The question is what kinds of science that would and wouldn’t be funded by private sources. Overwhelmingly, the kinds of science funded by nongovernmental sources tend to be R&D (e.g., pharmaceutical or technology companies, or lone inventors, doing research that can be directly translated into a product) or philanthropy-funded research, research funded by private charitable organizations (e.g. The Susan G. Komen Foundation, The March of Dimes, or other philanthropic organizations that fund research on a specific topic). So, yes, research would be funded without the government. It would tend to be much more “translational” and/or targeted at specific problems.
There’s also this dubious assertion by Kealey, cited approvingly by Ridley:
After all, in the late 19th and early 20th centuries, the U.S. and Britain made huge contributions to science with negligible public funding, while Germany and France, with hefty public funding, achieved no greater results either in science or in economics. After World War II, the U.S. and Britain began to fund science heavily from the public purse. With the success of war science and of Soviet state funding that led to Sputnik, it seemed obvious that state funding must make a difference.
Huh? In the late 19th and early 20th century Germany ruled physics, producing scientists like Max Planck, Albert Einstein, Werner Heisenberg, and many others, who made revolutionary discoveries that laid the groundwork for modern quantum physics. Germany was a powerhouse in science back then (and still is, only nowhere near as dominant). For example, the early 20th century, Germany won 14 out of first 31 Nobel Prizes in Chemistry. Just look at the Nobel Prizes in sciences! Until 1965, Germany won a larger percentage of science Nobel Prizes than any other country. I also note that the US didn’t catch up with France on that score until the 1940s. Obviously, Nobel Prizes are not in and of themselves an measure of how good a country is at science, but they do suggest where the most innovative research has been occurring a one to a few decades earlier, given that the science that wins Nobel Prizes is usually at least a decade old, to give time to see its significance. One can’t help but note that there is a correlation between the dominance of the US in Nobel Prizes and the start of government funding of science. Does correlation mean causation in this case? Not necessarily, given all the other factors that could impact this measure, but this observation is still a piece of data that at least calls Kealey’s assertion into serious doubt.
Basically, Ridley postulates the “myth” of basic science as a means of arguing that current patent policy is too stringent and protects monopoly (which is an arguable point) and that government funding “crowds out” private funding and prevents discoveries from being made:
To most people, the argument for public funding of science rests on a list of the discoveries made with public funds, from the Internet (defense science in the U.S.) to the Higgs boson (particle physics at CERN in Switzerland). But that is highly misleading. Given that government has funded science munificently from its huge tax take, it would be odd if it had not found out something. This tells us nothing about what would have been discovered by alternative funding arrangements.
And we can never know what discoveries were not made because government funding crowded out philanthropic and commercial funding, which might have had different priorities. In such an alternative world, it is highly unlikely that the great questions about life, the universe and the mind would have been neglected in favor of, say, how to clone rich people’s pets.
At this point, I gave up on Ridley. First, he’s downplaying the number of discoveries made with government funding, such as NIH and NSF funding. Also, the Internet is rather a big deal to dismiss so breezily as a “highly misleading” example, given how it has so thoroughly transformed our world over the last 25 years or so—mostly by private companies taking advantage of and building on the government-supported infrastructure and protocols. As for the last “what if” assertion, I did facepalm on that one, given that we actually do have a sort of “living experiment” going on right now regarding what happens when government funding dries up. The NIH budget has been more or less static for over a decade and thus has declined significantly in real dollars. As a result, private sources have stepped in. Have their priorities been better for the country? Not really:
Yet that personal setting of priorities is precisely what troubles some in the science establishment. Many of the patrons, they say, are ignoring basic research — the kind that investigates the riddles of nature and has produced centuries of breakthroughs, even whole industries — for a jumble of popular, feel-good fields like environmental studies and space exploration.
And:
Historically, disease research has been particularly prone to unequal attention along racial and economic lines. A look at major initiatives suggests that the philanthropists’ war on disease risks widening that gap, as a number of the campaigns, driven by personal adversity, target illnesses that predominantly afflict white people — like cystic fibrosis, melanoma and ovarian cancer.
A Nature editorial describes the problem well:
We applaud and fully support the injection of more private money into science, whether clinical or basic. Nevertheless, it is important for each funding body to take into account the kinds of research being heavily supported by the others, to avoid putting all our eggs into a few baskets and shortchanging areas that may yet have crucial contributions to make.
Ridley, given his seeming free market proclivities, might prefer market-based philanthropy to fund science over government funding (and that is his right), but he is sadly deluded if he thinks that private sources don’t “distort” scientific priorities every bit as much as he accuses government funding of doing and quite possibly even more. At least governments try to look at what will benefit the nation (or large parts of the nation); private philanthropists might or might not do that. Many simply respond to personal interests, personal tragedies, and, sometimes, crackpot ideas. Now it’s true that the government is by no means immune to crackpot ideas (witness the NCCIH, formerly known as NCCAM), but funding such ideas has to work within already established rules for peer review. The same is not true of private funding, where the philanthropist or foundation can basically make up any rules he or it likes to determine what research gets funded. Indeed, if he wishes, a philanthropist can even fund a scientist just because he takes a fancy to the work, no peer review needed.
In the end, I would argue that science should be funded by both government and private sources. What the optimal balance is will depend on the country, its priorities, and its economic resources. Contrary to what Ridley and Kealey argue, it doesn’t have to be a zero sum game. Even if Ridley is correct that technological discovery cannot be regulated by government (which he does argue with his likening of technology to a biological organism evolving), it would not follow that government should not fund research.
To see why, let’s revisit Ridley’s picture of technological progress as developing to become like a biological organism, complete with evolution in response to selective pressures. Now let’s carry that analogy farther than Ridley did. Just because evolution by natural selection still occurs in animals and plants doesn’t mean that selective breeding (i.e., guiding that evolution with human intent) doesn’t remain useful and effective in specific cases, such as breeding crops, dogs, horses, pigs, and other animals. Similarly, even if scientific and technological progress is evolving like species of organisms, that doesn’t mean that guiding certain the evolution of specific “species” of science by directing funding to human-decided priorities through government funding isn’t useful and effective.
In the end, all Ridley is arguing is that he prefers the science priorities of the private sector over priorities decided by government. To him one (private funding of science) is desirable and “natural,” while the other (government funding of science) is somehow “distorting” and therefore “unnatural” and wrong. Yet there is no real reason, other than personal preference, to view government funding of science as somehow less “natural” than private funding. To do so invokes an argument from antiquity, mainly that because private sources funded nearly all research until relatively recently that’s the “natural” state of things. Nor can Ridley convincingly show any fundamental difference in how much the whims of a few private donors and various industries “distort” science compared to government. In the end, Ridley just likes how private sources influence research priorities but doesn’t like how government influences them. That’s really all there is to his essay on science funding: Private sector good, government bad.

Lessons from the History of Insulin
Harriet Hall on November 6, 2012
Link
On my recent trip to Nashville for CSICon, I took advantage of the long hours on the plane to read Breakthrough: Elizabeth Hughes, the Discovery of Insulin, and the Making of a Medical Miracle, by Thea Cooper and Arthur Ainsberg. One of our commenters recommended it. I’m not sure who (was it Chris?), but I want to thank you.
It’s the history of insulin told from the perspective of the scientists and of a typical patient, and it touches on a number of issues that we have addressed on SBM. It shows how science works to save lives, in stark contrast to the empty promises of CAM.
The history
You would have to be a centenarian to remember what diabetes was like before the discovery of insulin in 1922. Type I diabetes in children was a death sentence. They wasted away, grew weak, and suffered indescribably before their inevitable death. They had insatiable thirst and hunger, but trying to satisfy their hunger only made things worse, and they continued to lose weight. The only treatment available was a rigorous diet (on the order of 400 calories a day with minimal carbohydrates), and all that did was prolong life by a few months. Patients usually had to be hospitalized to control their intake with carefully measured quantities of unpalatable food. Intake was adjusted by testing the urine, which was a complicated procedure at the time. Instead of a convenient dipstick, testing involved Benedict’s solution, test tubes, eyedroppers, teaspoons, a bottle for urine, and an aluminum cup. (Much later, urine testing was discarded in favor of the much more accurate blood testing.)
Earlier animal research had provided clues about the etiology of diabetes. They knew the pancreas was not producing adequate amounts of some unknown substance, and when aqueous extracts of pancreas were injected into diabetic dogs they appeared to have lower blood sugar levels; but research was interrupted by World War I. A Canadian doctor named Frederick Banting realized that the elusive substance was produced in the islets of Langerhans and that digestive secretions from the rest of the pancreas were destroying it during the extraction process. He devised elegant experiments in dogs, causing the pancreas to atrophy but leaving the islets intact. He succeeded in isolating the substance we now call insulin, and when he injected it into a pancreatectomized dog, all signs of diabetes disappeared.
The drug approval process
In today’s research environment, he could not have progressed to human experimentation as rapidly as he did. There was no time to lose: children were dying daily. They understood the mechanism and there was strong prior probability that insulin would save those children. There was little to lose, because without treatment those suffering children were as good as dead. So they rushed into treatment with small quantities of a poorly purified product that sometimes caused abscesses and reactions. Dosage was difficult to control, and patients sometimes went into hypoglycemic comas.
If the research were done today, it would qualify for the FDA Fast Track Development Program that accelerates approval of drugs that show promise for treating life-threatening diseases where there is no other good option.
The initial development of insulin took just under two years from the first dog experiments to large-scale production. The cost to the University of Toronto was $1,400, and Eli Lilly’s initial investment was $250,000. Today it typically takes 10-15 years to develop a new drug, and the cost often exceeds $1 billion.
Animal research is essential
It’s hard to imagine what would have happened if research on dogs had been prohibited. Insulin might not have been developed at all, or at least it would have taken decades longer. I think we can safely say that animal research saved many lives.
Big Pharma is not all bad
Insulin was in short supply until the Eli Lilly company got involved; they purchased truckloads of animal pancreases and developed a pure, standardized product. An anecdote in the book describes early problems with potency. The featured patient, Elizabeth Hughes, had to take 5cc at a time. Her syringe only held 2cc, so the needle had to be left in place while the glass syringe was unscrewed and refilled twice. The procedure took 20 minutes, and her hip swelled and her whole leg went numb. These symptoms resolved only after an hour of careful walking, but she was happy to accept all this in order to be able to eat real food again.
Eli Lilly was Big Pharma and of course they had a profit motive, but they were generous to those who couldn’t pay, and without their involvement the development process would have been vastly delayed and uncounted lives would have been lost. Lesson: Big Pharma is not the enemy: despite its drawbacks, it does a lot of good.
Genetic engineering is not all bad
Animal insulin caused reactions in some patients, so in 1982 Genentech marketed genetically engineered human insulin under the name Humulin. Almost all insulin used today is biosynthetic recombinant human insulin. How many of those people who are speaking out against the very idea of GMOs are diabetic, or have relatives who are diabetic?
The Nobel Prize and financial motivations
Banting and Macleod were awarded the Nobel Prize in 1923 for the practical extraction of insulin. They were incensed that the other members of their team were not included, and they immediately shared their prize money with Best and Collip. They sold the original patent to the University of Toronto for one half-dollar. They were not looking for fame or fortune; they wanted to keep sick children from dying. They did eventually benefit financially, but that was the last thing on their minds. The anti-vaccine folks have attacked Paul Offit for his role in developing the rotavirus vaccine; but I am convinced that he, too, acted out of concern for patients rather than from any desire to get rich. I am also convinced that many promoters of CAM care more about their wallets than their patients’ welfare.
Contrast insulin with CAM
Various natural treatments for diabetes are advocated by CAM enthusiasts:
    • Herbal products including bitter melon, cinnamon, fenugreek, ginseng, aloe vera, agaric mushroom, chromium, magnesium, prickly pear, soy, vanadium, oat bran, guar gum, glucomannan, bean pod, psyllium, white mulberry, alpha-lipoic acid, chia, coenzyme Q10, selenium, stevia, and others.
    • Homeopathy
    • Yoga
    • Acupressure
    • Reflexology
These are recommended mainly for type 2 diabetes, but also as adjuncts to insulin treatment in type 1 diabetes. In addition, many websites claim to cure type 2 diabetes. Diabetes can’t be cured, and evidence for clinical benefit from any of these natural remedies is weak or nonexistent. Even the usually credulous NCCAM warns:
In general, there is not enough scientific evidence to prove that dietary supplements have substantial benefits for type 2 diabetes or its complications… Do not replace scientifically proven treatments for diabetes with CAM treatments that are unproven.
Nevertheless, the NCCAM is funding studies of chromium, yoga, gingko, EGCG, Reiki, vitamin C, glucosamine, and dark chocolate for the treatment of diabetes.
Compare the contributions of CAM during the course of the 20th century with the accomplishments of scientific medicine. Science has conquered a disease that used to be uniformly fatal. It has given us disposable syringes, insulin pens, insulin pumps, pure insulin with measured potency, protamine zinc to slow the action, various long-lasting insulins, sulfated insulin to avoid insulin resistance, human insulin to avoid reactions, several effective oral medications for type 2 diabetes, convenient home blood testing gadgets, etc. And there have been innumerable studies to determine the best management of diabetic patients. CAM looks pretty puny in comparison.
We still don’t know what causes type 1 diabetes and we can’t prevent it; but we know what it is, and we can keep the disease under control with insulin, and we can keep patients alive and feeling well. This is not just “treating the symptoms” as our critics claim; and of course CAM does nothing to “treat the cause.”
The story of insulin is well worth reading. It’s a paean to science. It’s history with lessons for today.

Lessons from History of Medical Delusions
Brennen McKenzie on February 17, 2012
Link
A brief reference on the web site The Quackometer recently drew my attention to a very short book (really more of a pamphlet, in the historical sense) by Dr. Worthington Hooker, Lessons from the History of Medical Delusions, which I thought might be of interest to readers of this blog. Though published in 1850, the book contains many eloquent observations that are just as relevant to understanding how pseudoscience and quackery persist and even flourish in what we otherwise assume to be an age of scientific medicine. The book is available online as a Google eBook, and Dr. Hooker was a physician, a professor at Yale, and an outspoken critic of homeopathy in it’s early days. His critique of homeopathy still resonates today, and has long drawn the ire of Hahneman loyalists, such as this one who makes reference to Dr. Hooker’s, “periodical fulminations for the destruction of Homoeopathy that have appeared like locusts or cholera at certain dates.” Though Dr. Hooker wrote an entire book discussing homeopathy, Homeopathy: An Examination of its Doctrines and Evidences, he does spare a few words here for this less-than-venerated practice:
The error I have been illustrating is carried to an extreme by the Homeopathist. He attributes palpable results to doses of medicine which are so small that they cannot produce any perceptible effect except by miracle.
He also includes a lengthy and preposterous example of a homeopathic proving, taken from a homeopathic text of the time, illustrating the absurdity of simply listing every imaginable (and imagined) experience following the taking of a substance and then attributing the entire list to that substance in order to guide the selection and use of homeopathic remedies. However, the focus of this booklet is to illustrate more generally the sorts of errors in thinking that lead even otherwise intelligent and reasonable people to believe such nonsense.
And Hooker makes a specific point of reminding us that belief in medical absurdities is not by any means a characteristic only of the unintelligent, the uneducated or the past.
The history of medical delusions most copiously illustrates the truth, that folly is very far from being confined to fools.
The present generation laugh at the follies of the past but have quite as great follies of their own, an follies too of a similar character, and products of the same fundamental errors.
The majority [of believers in quackery] is made up of those who are more or less intelligent and rational on most subjects, but who…are especially deluded on the subject of medicine…The exposition I make is not a partial one. It is not a one-sided argument-a plea for the doctors against the people. But it is an attempt to show how both doctors and people have ever been liable to error, and how they have been alike in the common elements, if not in the forms and modes and fashions of their delusions.
The medical profession, like the community at large, is made up of fallible men, and the elements of delusion are the same in the one class as in the other [though] the error of the physician would be refined, and would have the pomp and circumstance of erudition.
Error gilded with the pomp and circumstance of erudition….That certainly brings a few names to mind, eh?
Some of the specific examples he uses are fine tidbits of historical minutia. Apparently, one of the founding fathers of chemistry, Bacon”>Francis Bacon, that luminary of critical thinking and scientific philosophy, advocated for applying healing salves to the weapon that made a wound rather than the wound itself (though given the loathsome nature of many therapeutic unguents of the time, this may not have been a bad idea since apply them to wounds doesn’t sound wise).
So what are the common “elements of delusion” that Hooker wishes to warn us of? He begins with the post hoc ergo propter hoc fallacy.
The first [element] which I shall notice is the too ready disposition to consider whatever follows as a cause as being the result of that cause.
He then points out the most obvious reason why this sort of reasoning so often misleads us in medicine:
The most important of the confounding causes is “vis medicatrix naturae, or the tendency there is in the system to remove disease and cure itself….there is in the system a tendency to spontaneous restoration in case of injury and disease…This tendency is the chief agency in most cases in curing disease. Sometimes it is the only one; and very often it effects a cure in spite of the mistaken and officious interference of art.
And yet quacks, and even physicians, and the public generally, are very prone to leave this agency out of view, and to attribute cures, as a matter of course, entirely to some favorite remedy which has been used. This disposition is the chief source of medical errors of all classes of men.
Hooker also touches on several other key sources of erroneous conclusions in evaluating medical theories, including confirmation bias, availability bias, anchoring, premature closure, sloppy use of analogous reasoning, passionate commitment to theories without empirical evidence, and medical fads, though all describe in a language rather more poetic than we would ordinarily use today.
He then goes on to talk about the issue of the commercial and political success of medical nonsense, which are certainly still relevant issues often discussed here.
So extensive is the popular delusion in regard to quack medicines, that the nostrum system has become an organized system, with an enormous machinery of certificates and advertisements. It has become a monstrous business interest, and is linked in with a thousand ties with other business interests. So powerful is it in this respect, that it has almost entirely subsidized the press, forcing it to be silent except when it speaks in it’s favor. The same may be substantially said when speaking of the action of legislatures on this subject.
Similarly, Hooker touches on the unfortunate aura of legitimacy that attaches to quack therapies when they are embraced by what he calls “medical men in good standing,” which could certainly be applied to the quackademic medicine phenomenon and the endorsement of medical nonsense by the likes of Dr. Oz and others.
Despite the eloquent expression of many issues associated with medical nonsense that are as relevant today as they were in 1850, not all of Dr. Hooker’s book will resonate with a modern audience. Apart from the florid prose style of the time, and the unabashedly sexist language, he scoffs a bit “the skeptic,” who he describes as sitting in “his ‘doubting castle’ well-fortified against all the shafts of truth.” He also was a fan of bloodletting as a remedy, and sneered at the research of Pierre Charles Alexandre Louis and others who demonstrated its lack of effect. In general, he was no fan of the “numerical” methods which have since developed into epidemiology, and he was overly respectful of the experience and judgment on individual doctors. Citing the same sloppy reasoning as is often used by modern proponents of alternative therapies, he argues that such “numerical observations…can be of no practical use to the physician in deciding in regard to any individual case…”
However, as a whole this little historical gem is strikingly applicable to the issues this blog deals with today. And it ends with a nice description of the gradual and imperfect process of vetting ideas through scientific inquiry, from initial unjustified enthusiasm to a gradual withering of bad ideas and a fitting of good ones into their appropriate but limited places.
While many remedies, once potent to cure in the public estimation, have….been wholly discarded, others, which have more real merit, while they have lost the extravagant reputation of their nascent state, have, under the watchful eye of experience, gradually obtained very nearly their right valuation, and the circumstances which should regulate their use have been ascertained with considerable accuracy. Others, in great numbers, are now going through this searching process; and others still are just now wearing the brilliant honors of an enthusiastic reception.
He also suggests, mistakenly I hope, that direct attacks on medical nonsense rarely have a salutary impact on the popularity of such practices. However, he also describes with some hopefulness the goal of his book, which I think to some extent describes the purpose of this blog as well.
No delusion however fiercely it may have been attacked was ever killed. Each after having withstood all assaults, has laid itself down o die in the most quiet manner, benumbed into the sleep of death by the chill of popular neglect, while the warm breeze of popular favor which it once enjoyed are now bestowed upon some other delusion…
And such exposition as this essay presents, of the common causes of medical delusion, both in the profession and in the community, will, I believe, commend itself to the reason and common sense of such persons, and will therefore have some influence, in connection with other kindred efforts, in deterring them from giving heir patronage to quackery in any form…

BBC Takes On Appeal to Nature Fallacy
Steven Novella on February 12, 2025
Link
It’s always good (and frustratingly rare) to see the mainstream media get it right when it comes to pseudoscience in medicine. Too often the narrative is – scientists are baffled at this alternative “one easy trick” to improve your health. Most mainstream articles on pseudoscience in medicine frame their reporting around a positive anecdote, and at best throw in some token skepticism as an afterthought, deep in the article that most readers won’t get to.
So I was happy to see an article in the BBC directly take on the appeal to nature fallacy and how it is used to market dubious health products. The article hits many of the notes that we have discussed often here. There is no clean definition of “natural”, and this vagueness is deliberately exploited by clever marketers. Further, there is absolutely no reason to think that substances or behaviors which occur in nature are better for human health than ones that are manufactured.
While the article was excellent as far as it went, I want to build on it a bit since I don’t think it fully captured the scope of this fallacy and its effects in our society. While the author, Amanda Ruggeri, correctly pointed out that this fallacy is hundreds of years old, the implications of this are worth exploring further.
The appeal to nature fallacy, the notion the products are inherently superior in direct proportion to how “natural” they are, has risen beyond marketing hype, and beyond mere propaganda. This has been a long term collective industry strategy that has altered society. In fact, research has shown that the “natural is better” bias is now the default in many areas – medicine, food, beauty products, lighting, even tobacco products (I will never forget the patient who reported smoking but reassured me that they only smoked organic cigarettes).
Rhetorically, the appeal to nature fallacy has also risen to a full-blown conspiracy theory in some forms. The narrative here is that nature has already provided everything we need to be perfectly healthy, but evil industry (Big Pharma, mainstream medicine, whatever) and their regulatory puppets have tried to deprive the public of the benefits of nature so they can sell more artificial treatments and products. The irony here is that it is the wellness/self-help/snake oil industry that is distorting reality for their own financial bottom line. One of their main champions, RFK Jr, is about to take charge of American healthcare.
This is more than just an innocent logical fallacy – this is a deliberate campaign of misinformation and disinformation, designed to defraud the public on a massive scale. The goal is to distract the public and their representatives from what really matters – what does the evidence say about the safety and effectiveness of any given product? But if you want to sell something that is unsafe and/or ineffective, you need to short circuit any process to determine these things objectively. So don’t look at the details, just be reassured and lulled into a false sense of security by the vague claim of being natural.
Often the promoters and sellers of such products just can’t be bothered with process. This is a major conflict in our society that even goes beyond science and medicine – we develop processes to determine what is fair, true, and reliable. These processes should be objective, transparent, and effective. The very concept of science-based medicine is all about process. The idea is that a good process will result in a good outcome.
But process is tedious, and doesn’t always result in an optimal outcome (because reality is complicated). Wouldn’t it be nice if we could bypass all that tedious, expensive, and complicated process an cut right to the outcome we desire? This is the ultimate appeal of the appeal to nature fallacy – substituting a simplistic algorithm (we like simplicity) for the complex process. Natural is good. Artificial is bad. To reinforce this algorithm we can also convince people that the process is corrupt, which is where the conspiracy theories come in.
Fully understood, the appeal to nature fallacy is not just a common mistake people make, it is a critical component of a cultural movement to dismantle the carefully constructed processes we have developed over more than a century to protect the public from fraud and harm. Why? Because some industries are inconvenienced by those processes. Specific industries target specific processes that are in their way, restricting their ability to defraud the public, extract resources, pollute the environment, change the climate, unleash dangerous but profitable products on the public, or whatever.
The wellness/alternative medicine industry (and this is a massive multi-billion dollar industry) leverages the appeal to nature fallacy and other strategies to relentlessly erode the processes we have developed to protect the public. They have tried to convince the public that they do not need protection, they just need unfettered access to all the natural cures that are out there.
This is the story I wish some mainstream outlets would tell. The public will lose tolerance for the tedious processes of science, research, review, transparency, and regulation if they don’t understand why we need them, and what the world was truly like before we had them. They may learn – the hard way – as we backslide into this pre-regulatory world where industry giants can foist their reality onto the public while saying – trust me, it’s natural.


   Critical Thinking: Medical Literature

   A Medical-Skeptical Classic
Mark Crislip
Link
The medical literature slowly becomes outdated. As a result there are not that many ‘classics’ in the field, since their content is becomes less relevant. The medical aphorism is that 10 years after graduation from medical school, half of everything you learned will no longer be valid. The problem for medical students is trying to figure out which half of their curriculum is not worth learning.
Old studies become increasingly irrelevant as diagnosis and treatment changes over time under the relentless pressure of medicine. I once came across the best of Osler, with his descriptions of typhoid fever and pneumococcal pneumonia. The essays were far more literary in style than todays journal articles, describing the presentation of these diseases in Dickens’ like detail, but of little practical help given the advances in treatment and the understanding of microbiology of diseases.
Technology also expands and limits what papers are available. If there is not an electronic form of an article, it might as well not exist. Many classic articles are not yet available in digital form, and the article in question for this post I had to get as a scanned version of the original paper, rather than a pdf. As a result of time and lack of electronic access, much of the older medical literature is not easily accessible, and journal publishers are not particularly interested in the free dissemination of information. Which is a shame. There is the occasional older references that is as applicable today as when it was published.
There was never, to the best of my recollection, a time when I was not a skeptic. But there was a time when I had neither the time nor the knowledge to be able to think skeptically about medicine. The torrent of information that needs to be assimilated in medical school, residency and the first part of fellowship makes reflection about that information almost impossible.
I do remember an article that was a turning point in my thinking about thinking about the practice of medicine. It was 1989 and the last year of my Fellowship and was published in the American Journal of Medicine, entitled “Observations on spiraling empiricism: its causes, allure, and perils, with particular reference to antibiotic therapy” by Kim and Gallis (hence forth called OOSE, pronounced, I suppose).
If you can scrounge up a copy, by all means do so, as it is a classic. It is a collection of logical fallacies and critical thinking as applied to infectious diseases. It was the first time I read an article that discussed how to think about thinking in medicine. I had no idea that there were logical fallacies. Most of what passed for critical thinking in my training concerned understanding the statistics and the materials & methods of studies. Important, but limited.
OOSE starts with a description of the discovery of antibiotics and the amazement of physicians that for the first time diseases that were often fatal were now curable. It must have been an amazing time for physicians when infections that routinely killed were suddenly vanquished. I have witnessed a similar revolution with the advent of HAART, where AIDS has gone from a 9 month life expectancy to a chronic disease with perhaps a normal life expectancy. At my hospital is Dr. Charles Grossman, now in his 90′s and still attending conferences. His long and productive medical career started at Boston Hospital where, as an intern, he was involved in giving the first iv dose of penicillin to a lady who was dying of a streptococcal infection, who survived another 50 years (2).
Antibiotics have been developed to kill increasingly resistant and virulent bacteria, and until recently we have managed to keep one step ahead of the organisms. Unfortunately, the ability to become resistant is outstripping out ability to develop new agents and we are slowly, and inexorably, losing the battle, sliding in to the post antibiotic era.
Part of the evolution of resistance occurs from the inappropriate use antibiotics, which can often be due to uncertainly of the diagnosis. But inappropriate antibiotic use can also due to faulty thinking.
As the authors of OOSE note, “The imprecision of clinical practice establishes context, the litigious nature of our society unnerves; the absence of toxicity permits; and the sum of these encourages the incontinent, extemporaneous use of these antimicrobial agent.”
Incontinent use of antibiotics leads to increasing bacterial resistance,and the use of antibiotics when they have not been needed has accelerated the evolution of pathogens, occasionally to the point where there are infections I cannot cure and bacteria that I cannot kill.
“The term spiraling empiricism describes the inappropriate treatment, or the unjustifiable escalation of treatment, of suspected but undocumented infectious diseases. Empiricism and empirical therapy, defined as the carefully considered, presumptive treatment of disease prior to the establishment of a diagnosis, often are necessary in the proper practice of medicine. On the other hand, ill-considered or inappropriate use of antibiotics, incurring unnecessary risk and expense, should be indicted and condemned, The difficulty like in distinguishing reasonable or appropriate from unreasonable or inappropriate therapy.”
As a teaching physician in a teaching hospital, I notice that sometimes it is the FUD, Fear Uncertainty and Doubt, combined with faulty thinking that sometimes leads to the inappropriate use of antibiotics.
OOSE provide a conceptual framework for approaching diseases and potential therapy (see table below); Observation, prophylaxis, empirical, therapeutic trial, and specific therapy. Of these, the first and the last, are, sadly, the least used. For interns and resident, the motto is ‘Don’t just stand there, do something’, and with the pressures to shorten hospital stay as much as possible, simply watching the patient is a luxury few can afford. As an experienced physician, I feel much more comfortable with the motto, ‘Don’t just do something, stand there’, or as the paper calls it “masterly inactivity.” As my wife can attest, I am the master of doing nothing.
Specific therapy in infectious diseases is not as common as I would like given the vagaries of growing the infecting organisms and the degree to which one wants to maximize diagnostic certainty. I could probably get the etiology of every pneumonia admitted to the hospital with an open lung biopsy, but it would hardly be worth the resultant morbidity and mortality.
Most of the time the patient is ill enough to be admitted to the hospital and, after appropriate studies and cultures are done, empiric therapy is started. That is often not an unreasonable course of action. These days you have to be ill to get admitted and it is the rare patient who comes into the hospital who can wait for cultures to be positive before beginning antibiotics. I cannot emphasize enough how ill patients are when they are admitted to the hospital, and how unclear the proper diagnosis can be at the beginning of a hospitalization. After a day or two all the diagnostic information has returned and the, with the 20:20 vision of hindsight, the correct diagnosis may become clear and as a result the proper course of therapy is clarified.
Since cultures are often negative, the empiric course of therapy may morphs into a therapeutic trial.
After setting the therapeutic framework in place, OOSE proceeds, with case reports, to describe fallacies in antibiotic therapy that lead to the wrong therapeutic interventions. There is, in medicine, a long tradition of using cases as illustrative of problems, but not as anecdotal evidence for the proof of a hypothesis. We all remember concepts when they are applied to specific patients and specific cases.
Their Fallacies in Antibiotic Therapy are
    • Broader is better.
    • Failure to respond is failure to cover.
    • When in doubt, change or add another.
    • Sickness requires immediate treatment.
    • Response implies diagnosis.
    • Bigger disease, bigger drugs.
    • Bigger disease, newer drugs.
    • Antibiotics are non-toxic.
I will add my own.
    • Once started, an antibiotic cannot be stopped.
    • Once a class of antibiotic is started, you need to stay in class.
    • Gotta double cover (i.e. give two antibiotics) a particular organism.
    • The primary reason a particular antibiotic is given is “I like it.”
I will not repeat the OOSE case studies as examples to support his fallacies, but instead will make comments and substitute my own examples. These fallacies are used a minority of the time, but it is not common to explicitly discuss the ways in which medical reasoning can go wrong in the context of faulty thinking. These fallacies are not unique to Infectious Diseases, and I am sure a neurologist or oncologic surgeon could generate a similar list in their own specialty. I doubt, however, that the same could be said of a chiropractor, naturopath, or homeopath. Perhaps one will prove me wrong in the comments, but logical fallacies are the sine qua non of alternative medicine. They are parallels and subsets of the standard logical fallacies beloved by skeptics everywhere. They are examples of the endless number of ways we can all think inaccurately.
Broader is better
Broad spectrum means an antibiotic that covers every conceivable pathogen. For most infections, the potential causes are often few in number, based on the history and potential exposures. You will always be surprised by the unexpected positive culture and any therapy will have gaps.
Residents do not want to be accused of not covering some particular organism if it turns out that one of the patients with an odd organism growing in their blood. So they give antibiotics that kill everything, adding expense and driving resistance without necessarily improving care.
Failure to respond is failure to cover
Few infections get better immediately, and some will be symptomatic for days. Staphylococcal endocarditis will have fever for up to 2 weeks, and often the reason a patient does not respond is not because they are on the wrong or inadequate antibiotic but it is either the natural history of the disease under treatment or, more often, there is a need to drain some pus. The old aphorisms “Nothing heals like cold steel” and “A chance to cut is a chance to cure,” remain as true today as they did in the pre antibiotic area. One of my colleagues does an impression of me when I am 85 and demented in a nursing home, banging my hand on the table, shouting over and over “drain the pus.”
Adding antibiotics may add only cost, potential toxicity and accelerate resistance.
When in doubt, change or add another antibiotic
Medicine is filled with uncertainty, and often it is the case that if an infection is thought of, it is treated, no matter how unlikely it is that it may be causing the disease. I have a fantasy where oncology is practiced like infectious disease. “It might be lymphoma, so lets start with CHOP, but we don’t want to miss adenocarcinoma, so lets add bleomycin, and in case its breast cancer we need to include tamoxifen and if could be prostrate so lets add etc , etc.”
When in doubt, increase your diagnostic certainty.
Sickness requires immediate treatment
Sometimes it does, sometimes it doesn’t. It depends on the patient and how clinically stable they are. All too often, people receive therapy for diseases that do not need antibiotics or the antibiotics are given before a firm diagnosis is made. This is driven, in part, by CMS core measurements. Community acquired pneumonia guidelines mandated that antibiotics be given within 4 hours of the presentation to the ER, if not the hospital gets a demerit. The result? Perhaps 1 in five patients who get treatment for pneumonia did not turn out to have pneumonia and would have benefited from watching and waiting (3). No good deed ever goes unpunished.
Healthcare providers appear to be particularly hesitant to not treat a fever with antibiotics. I like to say, antibiotics are not antipyretics, if you want to treat a fever, give Tylenol.
Response implies diagnosis
This is the most difficult fallacy for people to abandon. Patient has a fever, no diagnosis is evident, but the fever went away at some point after the institution of antibiotics.
Most fevers go away. Many diseases that are not infectious will have a fever. This is the medical equivalent of the skeptical motto ‘association is not causation’. The worst cognitive error physicians and patients make is the assumption, the Rigorously absence of a good diagnosis, that the improvement when a therapy is given is due to the therapy. Rigorously and consciously avoiding this error is key to being a good health care provider.
Bigger disease, bigger drugs
MRSA is a current bane of Infectious diseases. It is a bug that I cannot always kill with the current antibiotics, and the three or four antibiotics to which the organism is susceptible are all equally bad. So, when there is a severe MRSA pneumonia, the urge is to add rifampin, even though the data to support its use is problematic at best. But if one drug is good, are not two better? Perhaps. Sometimes combining antibiotics may be antagonistic and decrease the efficacy of therapy.
Bigger disease, newer drugs
Of course newer is better. New and improved. It is as American as Procter and Gamble. Not necessarily. For MSSA, there are no better drugs than nafcillin or cefazolin, and the newer agents are less efficacious in head to head trials. However, newer drugs are heavily advertised and detailed, <sarcasm> not that any physician would ever be swayed by advertisements instead of data </sarcasm>.
Antibiotics are non-toxic
All drugs have side effects, some common, some rare. Certainly antibiotics are less toxic than cancer chemotherapy agents, but they do have to potential to cause significant harm. We all have our irrational beliefs. Here is mine. Besides believing in critical thinking. The chance of an adverse reaction in inversely related to the need for the antibiotic. So if the patient has little need for the antibiotic, then the chance of an adverse effect is almost 100%. I know that it is not true, but some days the universe sure seems to conspire to function as if it were.
Some antibiotics are a big gun, are strong, or powerful
There are few things in medicine with 100% sensitivity and specificity. However, if your health care provider uses the adjectives big gun, strong, or powerful in reference to an antibiotic, they are either 1) talking out their backside or 2) ignorant about antibiotic use. 100% sensitive and specific
If I had neurosyphilis, and there are those who suspect I do, the ‘strong’ or ‘powerful’ ciprofloxicin would do nothing to treat my infection, put ancient, weak old penicillin remains the treatment of choice. What you want is to give are appropriate antibiotics: something that will reliably kill the organisms in whatever space is infected. These adjectives are advertising ploys used on fool gullible rubes, er, I mean health care providers, to think they are doing what is best for their patient. They provide a false security that you are giving the patient the best therapy.
Once started, an antibiotic cannot be stopped.
People have trouble stopping an antibiotic even when the cultures come back suggesting particular therapy is not needed. A patient comes in septic and antibiotics are begun. Even when it turns out that the blood grows a penicillin sensitive S. pneumoniae, the multiple initial empiric antibiotics may be continued. One of my colleagues only half jokingly says the reason she consults me is to have me stop the antibiotics. When there is reasonable data to suggest that shorter courses of antibiotics are no better than longer, the endless allure of 10, the number of fingers, or 14, twice two weeks, is often the default duration of therapy. I once saw a patient who had been on IV vancomycin for 6 months for osteomyelitis of the foot; the provider just couldn’t bring themselves to stop the therapy.
Once a class of antibiotic is started, you need to stay in class.
If a patient is admitted with an infection and started on an intravenous quinolone and grows something than can be treated with a less expensive, narrower spectrum agent like amoxicillin, at discharge the patient is likely to go home on an expense oral quinolone. They responded to a class of antibiotics, and so it needs to be continued, even if the cultures say otherwise. Magical thinking, as best I can tell, and one that I occasionally participate in.
Gotta double cover (i.e. give two antibiotics) a particular organism
There is a persisting myth that some infections, like Pseudomonas, always require two antibiotics. That is true occasionally, but for most cases a single antibacterial is no worse than two, and the more antibiotics you give the more you increase the toxicity’s and the faster you drive antibiotic resistance.
The primary reason a particular antibiotic is given is that “I like it.”
I like to say that the three most dangerous words in medicine, especially when it comes to therapeutic interventions, are “In my experience.” In my experience you can’t trust anyone who uses that phrase. Remembering hits and forgetting misses drives antibiotic use more than I would like to admit. Infectious disease docs are sometimes the opposite. We put too much emphasis on the antibiotics that have failed us in the past. It is o e extreme or the other.
Patient is admitted with a complicated infection and started on X. Why X? I ask. It worked for me in the past is the reply. What are you trying to kill? I might then ask. Often, they say the boogie man of the ICU, Pseudomonas. What I will continue, is the chance drug X will be effective against Pseudomonas? They don’t know. So why again are you using X? It is what we do. They have used it successfully in the past, so it should work again.
Sigh.
Fortunately, most drugs work most of the time in most patients, but that rule is slowly being lost as the organisms become increasingly resistant to our current armamentarium of antibiotics and there are few, if any, replacements in the pipeline. At my institution there is 2% resistance for Pseudomonas to Ceftazadime and 4% to pipercillin/tazobactam. We get a few cases a year at best of a blood stream infection and sepsis from Pseudomonas. Most physicians are not going to see enough infections by a given organism to get a sense of what does and does not work. It is why you cannot trust your experience.
I have not yet read the book “How Doctors Think” although being in a teaching hospital I try, however poorly, to demonstrate how I think about thinking about infections. I often give the residents on service a copy of the article on which this post is based. Once, when I was ranting about this at a conference, one of the attendees said I was an arrogant subspecialty who didn’t understand the pressures faced by general internists.
I don’t know. Perhaps they are right. But often getting the right diagnosis and therapy is less about what you know and more about being rigorous about understanding how you know. Only when you are conscious of your ability to think poorly, can you compensate. In the era of Google, any knowledge gap is a search away, but you have to be aware of the knowledge gap first.
To finish quoting this under-read and under-appreciated article
“…fashion and marketing forces, not science, shape clinical practice. Within an intensive are unit, gravity of illness amplifies the effects of delay and presses for preemptive, urgent empiricism, but the selection and manipulation of therapy can sometimes be justly criticized. The intensive care unit mentality, which treats all illness as life-threatening, favors therapy over careful consideration of diagnostic maneuvers… The potential for, and tendency toward, spiraling (as opposed to appropriate) empiricism appears disturbingly pervasive- reinforced by fashion, compensation, and litigation.”
True, but driven more by the urge to help people get better and the fear of doing harm by missing an infection. OOSE does overstate the problem, but such is the nature of polemics.
“Osler’s comment that the ‘practice of medicine is an art, based on science’ underscores the dichotomy and the dilemma, of medical science. Empiric therapy at its best is set between the Scylla of unnecessary delay and the Charybdis of therapeutic voyeurism. Knowledge of the science of medicine and the natural history of disease should temper and compliment the art.”
This article was one of the few epiphanies in my medical career. I re-read it several times a year and consciously endeavor to keep it’s concepts in the front of my thought processes. It stays fresh with each reading, because my capacity to think badly is limitless and only by constantly reminding myself of that fact can I hope not to stray into illogic.
Part of applying the art of medicine is to recognize the fallacies of medical thinking and to question your decisions and interventions in the light of understanding yourself and how you think.
The bumper sticker “Don’t believe everything you think” should be a motto for all health care providers.
References
Am J Med. 1989 Aug;87(2):201-6. Observations on spiraling empiricism: its causes, allure, and perils, with particular reference to antibiotic therapy.
The first use of penicillin in the United States. Grossman CM. Ann Intern Med. 2008 Jul 15;149(2):135-6. PMID: 18626052
Public Reporting of Antibiotic Timing in Patients with Pneumonia: Lessons from a Flawed Performance Measure. Ann Intern Med. 2008;149:29-32.

   SOS DD
Mark Crislip 
Link
What does it take to become a doctor? Endurance and perseverance help. It is a long haul from college to practice. But the skill that is most beneficial is the ability to consume prodigious amounts of information, remember it, and recall it as needed. Although I often relied on ‘B’ to get me through some of the exams.
Thinking, specifically critical thinking, is not high on the list of abilities that are needed to become or be a doctor. Day to day, doctors need to think clinically, not critically. Clinical thinking consists of synthesizing the history, the physical and the diagnostic studies and deciding upon a diagnosis and a treatment plan. It is not as simple as you might think. When medical students start their clinical rotations and you read their notes, you realize they have what amounts to an advanced degree at Google U. They know a huge amount of information, but have no idea how the information interrelates and how to apply the that information to a specific clinical scenario. With time and experience, and it takes at least a decade, students become clinicians and master how think clinically, but rarely the need to think critically.
The volume of data combined with time constraints ensures that we need to rely on the medical hierarchy to help manage the information overload required to apply science and evidence based therapy. There is just to much data for one tiny brain to consume. Other doctors rely on me for the diagnosis and treatment of odd infections. In turn, I rely on the published knowledge and experience of my colleagues who have devoted a career to one aspect of infectious diseases. There is little time for most doctors to read all the medical literature carefully, and usually little need. We have people and institutions we use as surrogates.
Not only is critical thinking usually not required to be a good physician, but medical practice can conspire to give physicians a false sense of their own abilities. Really. Some doctors have an inflated sense of self worth. Who would have thought it? Spend time with some doctors and listen to them pontificate on politics or economics with the same (false) assurance that have in their true field of expertise, and you will run screaming from the room.
Medicine is rife with confirmation error. In practice there is rarely negative feedback for making a wrong diagnosis, since everyone is wrong all the time. The good doctors are nimble, changing course quickly rather than sticking to a wrong diagnosis. And there is no lack of positive feedback from good results. There is the potential for unbalanced positive feedback to engorge the susceptible ego. Last week I was walking downtown and was stopped by a lady, daughter of a patient, who introduced me to her friends as “the doctor who saved my mothers life.” Not really true given that medicine is a team effort, and while I made the diagnosis others had missed, I know if it I had not made the diagnosis, some other ID doctor would have. Talk about a chance for egoboo. On the other hand, I must admit to great irritation when the hard work of the ICU team pulls someones bacon out of the fire and the credit goes to god. I want to yell it was not god that saved your father, it was the doctors and nurses and respiratory therapists and the rest of the hard working, highly experienced ICU staff. And just why did god make him so ill in the first place? I do want at least partial credit and remain grateful that mind reading is not a reality.
The other weird thing that happens in medicine is the explicit knowledge we learn in medical school and residency over time becomes implicit. I have discussed before how sometimes I can come to the correct diagnosis with minimal information, almost intuition, and then explain after the fact how I came to the diagnosis. It almost feels like the answer bubbles up from below awareness. It is most peculiar how much of my medical reasoning occurs below my consciousness. If one is not careful that seeming quasi-omniscience could lead to a certain arrogance about your own abilities. And if part of your job is to hold peoples living hearts in your hand? It amazes me that the heart surgeons I work with are such nice, self-effacing people.
Add the lack of training in critical thinking to the weird combination of the Peter Principle and Dunning-Kruger Effect and you have a SCAM proponent in the making. The first says that people rise to their level of incompetence, and if someone masters an area of expertise, they become incompetent in another field. The latter suggests that “people make poor decisions and reach erroneous conclusions, but their incompetence denies them the metacognitive ability to appreciate their mistakes.” When good docs go SCAM.
Starting with the basic sciences as a premed and working your way through the Brobdingnagian quantities of information to understand the workings of the human body in health and disease results in knowledge, but not necessarily wisdom; learning not how to think, but what to think. You have to trust that those up and down the line of experience know, at least in part, what they are doing: intern to resident to chief resident to attending to subspecialist. I variously use team sport and hive intelligence to describe the practice of hospital medicine, although the Borg may be a better metaphor. That being said, there is, at least in my small neck of the woods, little dogmatism. Skeptics, die hard and otherwise, are often portrayed as close-minded and dogmatic, but in the teaching hospitals I practice in, explanations of the medical literature are often filled with qualifications and caveats as we teach the residents. How much of the great grey uncertainty that is medicine is passed on to patients is always an interesting question, and different clinicians will react differently to that clinical challenge.
I rarely see the caveats and uncertainty in the SCAM world that is common in medicine. Often my answer to a consult is that the patient does not have an infection, or if they do, I have no specific therapy and you have to let the disease run its course. I wonder, in passing, if anyone has ever gone to a SCAM provider and been told: Nope. Sorry. Can’t help you. —Insert magic here— has no benefit for —insert disease here—. I bet almost never.
Patients use SCAMs for a variety of reasons, and I do not pretend to understand why people prefer magic over reality. And, sounding like the arrogant tool of the medical industrial complex that I am, I wonder if part of the reason is they don’t know any better, and, really, most people have no reason to. The average person has little motive or inclination to become immersed in the background required to understand why SCAMs are nonsense. They have better things to do.
But MD’s? You would think a doctor would know better. Astronomers do not flock to astrology. Physicist do not push perpetual motion machines in large numbers. Geologists do not become young Earth creationists in droves. It flabbers my gaster that, after learning all the information required to put MD after your name, anyone could find any validity in any SCAM. Yet many do, too many. Part of my assumption is that the Oz’s and Chopra’s and Weil’s of the world do not process information about the world in the same way the Gorski’s and Novella’s and Atwood’s do. Either they never learned, or cannot learn, critical thinking, or, most importantly, applying the concepts to themselves.
I was originally dragging my feet writing this essay, as the topic was going to be Dr. Oz, who consistently demonstrates all the cognitive flaws and biases mentioned above. The first thing I did when I received my medical school acceptance letter, besides checking the name to make sure it was really addressed to me and not a mistake, was sell my electric typewriter, thinking I would never have to type again. The thought of transcribing his shows filled me with dread, as my typing skills are barely up to the task to transcribing the thoughts in my head, much less the words that flow from Dr. Oz. Imagine my delight when I came across an editorial in JAMA, Medicine’s Great Divide—The View from the Alternative Side, by Dr. Chopra, which has the same odd world view and odd thinking with none of the transcription issues. Cut and paste is so much easier.
The world of Dr. Chopra is radically different from the world in which I live and practice medicine. My professional world view is equal parts an understanding of objective science and an understanding of the almost too numerous to count human intellectual and physical frailties and short comings that make applying science to patient care difficult and prone to error. And that doesn’t even include just how hard it is to figure out what is going on in patients. Of course, people never see themselves as other see them. When I read Chopra’s account, I also have to ask myself, is my professional world view correct. Or am I just this guy, you know? But my Beeblebrox-like internal monologue reassuringly responds no. Let us go todash.
Conventional medicine is offended that alternative medicine even exists. For the average physician, to hear that an allergy patient is taking extract of nettle to treat his symptoms or that a breast cancer patient is being treated with coffee enemas and a macrobiotic diet arouses scorn. Over a decade ago, when the New England Journal of Medicine reported that Americans pay more visits annually to alternative practitioners than to MDs, the attitude of the editorial writer was barely disguised dismay and disbelief. It was as if the whole country had turned its back on jet travel to return to the horse and buggy.
I do not speak for conventional medicine, or as I like to call it, medicine, only for myself, but I do have a sense of dismay when doctors use and promote SCAMs. Other people? Not so much. Dr. Chopra evidently feels that relying on the irrational magics of earlier ages to be a reasonable way to approach health and disease. It is worse than horse and buggy, which will at least get you to your destination. SCAMs are the intellectual equivalent of returning to demon exorcism, cupping, bleeding and purging to treat and prevent illness. It is applying the principles of Harry Potter in a world where disease and death are real. In my world view, a bad idea. On the other hand, in the entire essay, he never once mentions a specific wackaloon SCAM as worthy for use in patient care, preferring to focus on the shortcomings of medicine.
Yet at bottom no one could really object to the aims of alternative medicine, which are to bring relief to the whole patient.
What is that old road to hell paved with? I can’t recall. Object to the aims? No. Everyone thinks they are acting in a beneficent manner he writes resisting the urge to be accused of Godwins Law. The methods? Well, yeah, I object to the methods, The objections with SCAMs have nothing to do with aims, Dr. Strawman. It is with how those aims are accomplished. I have trouble trusting peoples health and well being with interventions that have no basis in my understanding of reality.
He spends the rest of the paragraph dwelling on the pathologies of modern medicine: the lack of time and the unpleasant environments of modern hospitals, with the usual implied argument that since medicine has issues, SCAMs must be valid and the solution.
In other words, the other party in the divorce—those who have lost faith in conventional medicine—has its own valid reasons.
Because I am fat, therefore you are thin. It gets tiresome seeing the same flawed arguments. Has Dr. Chopra never bothered to peruse the cognitive error or logical fallacies pages on the Wikipedia to at least casually educate himself on the bare minimum of critical thinking? SCAMs of all stripe have to have to stand or fall on their own merits, not on the perceived and real failings of medicine. The response of my hospitals to the problems with medicine has not been to abandon medical science in favor of nonsense, but to rigorously apply the results improve on what we have, with remarkable results. We have made the airplane fly better, safer and more comfortably, not abandoned the airplane in favor of levitation. Italics mine:
The two camps are not as opposed as they once were. Twenty-five years ago the possible efficacy of traditional healing modalities, herbs, Eastern therapies like acupuncture, and even mind-body medicine was so foreign as to be entirely alien. Today there are still die-hard skeptics (i.e. people with a consistent understanding of reality who understand that the shortcomings of human cognition apply to themselves as much as to others), of course. But in a mood of expanded tolerance (read easy money), an MD can look at the research on neurotransmitters, cell membrane receptors, and brain physiology, which has made enormous strides in recent decades. Taken as a whole, this research describes the body as an integrated system that exchanges information continuously between the mind, via the brain, and every cell in the body.
What? Huh? Say what? I can’t see how the first part of the paragraph relates to the second half of the paragraph. I have to admit, with a child in middle-school, I have been reading my sons five-paragraph essays, with their emphasis on logical flow and coherence, so perhaps I am out of practice with reading essays written by adults and miss the subtle connections between alien modalities and modern neurophysiology.
One aspect of SCAM proponents I note is they like to take obvious conclusions, what should be almost banal statements, and make them seem like they are profound insights into the human condition available only to SCAM proponents, “the body as an integrated system that exchanges information continuously between the mind, via the brain, and every cell in the body.” Well, duh.
I guess Dr. Chopra hasn’t spent time on rounds in the hospital for a while, where the endless permutations of physiologic interactions and complexity are discussed, observed and modified. Understanding these interactions is part of what makes the practice of medicine, real medicine, so damn hard. It must be so much easier to do some quantum mind-body hand waving and not have to actually fret about the application of practical physiology.
With real-time scans from functional MRIs staring them in the face, MD’s have no reason to look upon the placebo effect, for example, as “not real medicine.” When patients feel relief from chronic pain by being given a sugar pill, the body’s endorphins are filling the same receptor sites in the brain that externally administered opiates fill.
Placebo has always been real medicine, the question is whether using a placebo is ethical (it is not, since lying to a patient is not ethical) and how effective placebo is (for pain, barely efficacious, for any other disease placebo has no effect). In my world view placebo is wrong, because lying to patients is wrong, its effects are marginal (especially for infections) and I have yet to find a way to use placebo without lying to a patient.
But SCAM practitioners have a key insight that only they have access to: the importance of diet. Of course he transitions into the old trope that diet is alternative. Alternative to starving I suppose. As my son would say to me, how edgy. Changing diet can improve cardiovascular disease. <sarcasm>WTF. No one ever told my ever in the last 25 years that diet was important for cardiovascular and other diseases. I want a refund from OHSU.</sarcasm>. It is important for SCAM practitioners to pretend that diet is somehow alternative, and then by extension, validating real nonsense in fields like reiki or acupuncture or homeopathy. SOS DD.
Seriously, what offends me is not alternative medicine, but sloppy, inaccurate, lazy thinking, be it in SCAMs, politics or literary criticism. And so much of SCAM is sloppy thinking, sometimes from educated people who I think should know better. But I remain of the opinion that sloppy thinkers make sloppy doctors. Dr. Chopra is a board-certified endocrinologist, and he is trying to imply with a straight face that lifestyle changes, diet and exercise, are not at the core of the approach to type 2 diabetes and cardiovascular disease? Ground rounds this week was an update on hypertension. The first part of the lecture? Diet and exercise. It is the old truth that if you repeat a lie often enough, people will come to think it true.
One sign of growing reconciliation comes in the form of softened terminology. Instead of calling it alternative or holistic medicine, as I’ve been doing, the more acceptable term is complementary and alternative medicine (CAM), which sends the signal, “See? I am not your foe. We can cooperate. We’re complementary.” Which is true.
It reads like Jake Chambers final essay. You have to watch CAM all the time, CAM is a pain and that is the truth. Really? You can add nothing to something and increase its value? If the integrative medical centers ever go under, they may have a job on Wall Street. 1 plus 0 is not 2, except in the world of SCAM. See. I am your foe. You offer nothing of value. Which is true.
Then he shifts to reality mixed with BS. 
The problem is that an MD’s practice is badly set up to promote prevention (often true). Visits are too short (yep). Aren’t adequately trained beyond their specializations (BS). Their habits are focused almost entirely on drugs and surgery as treatment modalities (BS). Prevention is considered too “soft,” (BS) and yet, if you shift the burden of prevention to the patient (lets call it empowerment when the SCAMsters do it) (which most MDs are more than happy to do (lets call it a partnership when the SCAMsters do it)), there is enormous resistance (BS). The public has been given countless warnings about smoking, poor diet, and lack of exercise, yet we have by no means eradicated lung cancer, obesity, coronary artery disease, and type II diabetes (true). Lifestyle disorders prove intractable when people cling to bad lifestyle habits and resist adopting good ones (true, but whose ‘fault’ is that?). We remain a nation of sedentary overeaters, paying pious lip service to prevention while doing less than enough about it (true, but is the solution in SCAM?).
This is where CAM makes significant inroads, because one of its main themes is the return of power to the patient.
See, I told you where this was going. Doctors dump, CAM returns power to the patient. I love the language of BS. The lessons of Animal Farm and 1984 remain relevant in 2011, and probably for the life of the human species. I imagine that those who seek SCAMs are motivated to see a benefit from a particular SCAM. SCAM providers see a self-selected population who want to have results from what ever modality they are using. I would wonder how effective any SCAM based therapy would work if tested on a population of CFI or JREF members. So much potential for confirmation bias in the assertion that SCAMs have a unique ability to alter peoples behavior by empowering the patient.
MDs should welcome the whole trend to self-treatment instead of taking the scornful attitude that nothing works but the modalities taught in medical school.
So when is shifting the burden magically transformed into a trend to self treatment? When it is offered by a SCAM practitioner. Most MDs are aware that medical school is the beginning, not the end, of learning, and also recognize that it is as important to know what isn’t effective is as important as knowing what is effective. It is why medicine evolves and SCAMS are intellectually stagnant. Dr. Chopra seems to invent a world that does not exist, at least in Portland, and then rail against it. Trying to engage the patient is their care has been a goal in medicine for decades.
The real mystery—one that deeply intrigued me 25 years ago—is that so many therapies that totally disagree with one another manage to bring results. Ayurveda isn’t qigong; yoga isn’t Reiki; none of them is a placebo. Yet somehow healing exists, and the channel it takes can be quite unexpected and inexplicable.
The real mystery is that so many forms of astrology totally disagree with another manage to bring results. No mystery if you live in a reality-based environment. The real answer, the rational answer, is that none of them do diddley. People often get better and always have and the ability for people to fool themselves and others is endless. That paragraph, more than any other in the essay, sums up the difference I see between SCAM proponents and SBM: a complete lack of insight and critical thinking that allows people to not recognize their own limitations in understanding the world. Yet I am the closed minded arrogant tool. Sheesh.
Continuing in a vein separate from the world I work in,
The inconvenient truth that “you can heal yourself” has always been the foundation of medicine. The body is the locus of the healing system; physicians assist this complex, little-understood system. They do not actually do the healing. If this feels threatening to MDs, there is much more room for pride to take a fall.
Dude, its not inconvenient at all. We all know that. No one is threatened. So many Stuffy Guys, I worry about a tremendous conflagration should nonsense arguments become flammable. What’s next? We in SCAM understand that water is wet and fire is hot? Only SCAM practitioners seem to make it a virtue to trumpet the obvious as if it were some profound truth. Little understood? What, you give up reading the literature? Understanding of healing is not complete, but it is broad and deep.
Conventional medicine also faces the mysterious “decline effect”—established medications steadily lose their effectiveness over time, as if the newer generation of patients has different, less receptive physiologies.
Is this willful ignorance? Massive confirmation bias? Mysterious? Really? Doesn’t he read and think about these topics? Dr. Steven Novella:
…it is no surprise that effect science in preliminary studies tend to be positive. This can be explained on the basis of experimenter bias — scientists want to find positive results, and initial experiments are often flawed or less than rigorous. It takes time to figure out how to rigorously study a question, and so early studies will tend not to control for all the necessary variables. There is further publication bias in which positive studies tend to be published more than negative studies.
Further, some preliminary research may be based upon chance observations — a false pattern based upon a quirky cluster of events. If these initial observations are used in the preliminary studies, then the statistical fluke will be carried forward. Later studies are then likely to exhibit a regression to the mean, or a return to more statistically likely results (which is exactly why you shouldn’t use initial data when replicating a result, but should use entirely fresh data – a mistake for which astrologers are infamous).
Are SCAMs immune from the decline effect? Probably, since you can’t decline past zero. Any further decline in the effects of homeopathy or acupuncture or energy therapy would soon cross into increasing the symptoms they are attempting to treat.
For me, the most distressing aspect of the decline effect is how widely it is being ignored.
Come on rounds, Dr. Chopra. Listen to our Intensivists discuss the use of steroids in sepsis or the Cardiologist discuss the approach to acute coronary events with the residents. The history of the literature, why the approach has evolved as the studies have refined and extended our understanding. The decline effect is an integral part of the teaching of medicine, it is an integral part of the history of science and the evolution medicine. Just like the complexity and the integrated systems of the body, it is part of the foundation of the practice of medicine, although, like much of the foundations of medicine, rarely explicitly stated, except as a source of pseudo-profundity.
More distressing appears to be the inability to apply the same worry to SCAMs, where the history demonstrated increasingly better studies of SCAMs resulting in decreasing efficacy until well designed, bias free studies show no efficacy. Not a decline effect, but a crash, burn and explode effect, always ignored by the SCAM proponents. Oh well, a foolish inconsistency is adored by SCAM minds and its philosophizers. If you think medicine has issues, read the CAM literature. It is nice how artfully Dr Chopra fails to note that the literature in the SCAM world is several logs worse and almost no intervention has any physical or biological plausibility.
My purpose isn’t to justify the vast universe of healing modalities that exist outside Harvard Medical School (Besides the Palin-esque stab at populist anti-intellectualism, I note that Harvard is long past it’s prime as a medical arbitrator, what with the Osher Clinical Center and all. Harvard has consumed the Kool-aid). I look instead toward the next phase of this reverse marriage, which is shy courtship. If both sides stopped being defensive, they would see that they share core values: treating the whole patient, reducing suffering, closing the gap between healer and healed, and doing the least harm while bringing the greatest good.
Whoops. Forgot a core value: trusting patients lives and fortunes to therapies that have efficacy based on reality and that are effective. Small thing, I know, but I am old school and feel obligated to treat patients with therapies that are effective beyond the self delusions and biases of the practitioner.
With that in mind, I have little desire to debate with skeptics and scientists who disdain CAM and falsely claim that only their side is valid and evidence-based. The mystery of healing remains unsolved.
Whew. I guess he will not be responding in the comments and prefers to live in his own world. Like people who only watch Fox news. This week I helped cure/heal two heart valve infections, a relapsing fever, a malaria and a handful of pneumonias and skin infections. No mystery there for me. Do I understand all parts of healing? Nope. Does that invalidate the parts I do understand? Nope. Would any SCAM have ‘complemented’ any part of my patients course? Nope. I do agree that SCAMs are evidence-based, it is just that the science and evidence demonstrates they do nothing and that science and evidence has validated their worthlessness.
He concludes
The next step will be to remodel medical school curricula so that future physicians are not wandering in the dark as my generation did, totally ignorant, if not blind, about treatments outside our narrow band of knowledge. Expanded medicine is the answer, I am sure of that.
Just as I am sure the world will not be improved by SCAMs modalities. For fun, let us make a list on the top 10 advances in human health made in the last 100 years, in the US or elsewhere, due to application of any SCAM:
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
My wife likes to say if you want to predict the future, look at the past.
I do not practice in the world of Dr. Chopra or understand its allure. His medical world is divorced from the one I work in, and his approach to understanding medicine could be used as a textbook examples on logical fallacies, cognitive errors, and disingenuous argument. I am less than impressed. As Dr. Feynman said, “The first principle is that you must not fool yourself, and you are the easiest person to fool.” Unless you think the future is expanded medicine.
SOS DD: same old stuff, different Doc. It has other meanings.

   Cochrane is Starting to ‘Get’ SBM!
Kimball Atwood
Link
This essay is the latest in the series indexed at the bottom.* It follows several (nos. 10-14) that responded to a critique by statistician Stephen Simon, who had taken issue with our asserting an important distinction between Science-Based Medicine (SBM) and Evidence-Based Medicine (EBM). (Dr. Gorski also posted a response to Dr. Simon’s critique). A quick-if-incomplete Review can be found here.
One of Dr. Simon’s points was this:
I am as harshly critical of the hierarchy of evidence as anyone. I see this as something that will self-correct over time, and I see people within EBM working both formally and informally to replace the rigid hierarchy with something that places each research study in context. I’m staying with EBM because I believe that people who practice EBM thoughtfully do consider mechanisms carefully. That includes the Cochrane Collaboration.
To which I responded:
We don’t see much evidence that people at the highest levels of EBM, eg, Sackett’s Center for EBM or Cochrane, are “working both formally and informally to replace the rigid hierarchy with something that places each research study in context.”
Hallafrickin’loo-ya
Well, perhaps I shouldn’t have been so quick to quip—or perhaps that was exactly what the doctor ordered, as will become clear—because on March 5th, nearly four months after writing those words, I received this email from Karianne Hammerstrøm, the Trials Search Coordinator and Managing Editor for The Campbell Collaboration, which lists Cochrane as one of its partners and which, together with the Norwegian Knowledge Centre for the Health Services, is a source of systematic reviews:
I just wanted you to let you know that I have been playing around with the same thoughts as you express in the EBM/SBM Redux series; having come across related problems in other reviews and finding the laetrile review by chance – as well as following the SBM blog (strangely enough I corresponded with Dr. Ernst concerning laetrile the day before you posted your correspondence with him – he must be getting tired of these e-mails!). For this reason a colleague and I wrote a letter to Cochrane, a letter which they have, to my surprise, accepted as an editorial and which will be published in mid March, I believe (The SBM blog is duly credited). Just wanted to let you know, and also that the response from Cochrane has been overwhelmingly positive.
Thanks for a very interesting, entertaining and educating blog!
Well, with no small sense of self-satisfaction I thanked her and forwarded her email to the other authors here, who got a kick out of it, but then I kinda forgot about it until trusty SBM commenter Peter Moran posted a link to the promised editorial. Lo and behold, woodja look at the very first sentence! Its citation is the post in which I’d dismissed Dr. Simon’s assertion about Cochrane placing research studies in context, and in which I reported my correspondence with Dr. Ernst regarding the Cochrane Laetrile review. But yes, I may have been a bit too facile in my dismissal of Prof. Simon’s contention, because it’s clear from the editorial and its ‘feedback’ that others, even among Cochrane reviewers themselves, have been similarly bothered. The problem, elsewhere dubbed EBM’s ‘scientific blind spot,’ nevertheless remains the rule rather than the exception. Two of the three feedback letters that are available as of this writing, moreover, don’t fully grasp the point.
Those of you who’ve been following this series know that I’ve already mentioned an exception to the EBM scientific blind spot at Cochrane, regarding its Laetrile review. It’s found not in the review itself, but in the form of Feedback from another Cochrane reviewer, who made arguments similar to my own. Today I’ll discuss another exception, the best that I’ve found so far, and for the second time today I’ll tip my hat to Scandinavians.
Intercessory Prayer
A 2009 revision of “Intercessory prayer for the alleviation of ill health” begins as follows:
This revised version of the review has been prepared in response to feedback and to reﬂect new methods in the conduct and presentation of Cochrane reviews.
There are interesting changes in this revision, some of them having to do with what we’ve been talking about. Let’s go right to the punch line. The first sentence is old hat; the second is nearly revolutionary for EBM:
Authors’ conclusions
These ﬁndings are equivocal and, although some of the results of individual studies suggest a positive effect of intercessory prayer, the majority do not and the evidence does not support a recommendation either in favour or against the use of intercessory prayer. We are not convinced that further trials of this intervention should be undertaken and would prefer to see any resources available for such a trial used to investigate other questions in health care.
Wow! Previous iterations of this review, spanning about a decade, had made the customary call for “further study.” What changed? Don’t get too excited, even though I’ve been goading you: what the authors left unstated were their reasons, intuitive though they might have been, for not being “convinced that further trials should be undertaken.” That, I suppose, would’ve been just too dicey.
Before discussing what actually changed, let me explain a couple of key features of this review. First, the authors take pains to acknowledge but, er, distance themselves from religious implications:
How the intervention might work
The mechanism(s) by which prayer might work is unknown and hypotheses about this will depend to a large extent on religious beliefs. This review seeks to answer the question of effect not mechanism and it does not seek to answer the question of whether any effects of prayer conﬁrm or refute the existence of God…
…the results of this review will be of interest to those who are involved with the ‘debate about God’ – both religious believers and atheists – but these results cannot directly stand as ‘proof ’ or ‘disproof ’ of the existence of God…We do not, therefore, seek to pose or answer any questions about the existence of God with this review.
They observe that there are “several challenges” in performing such trials, naming “contamination” (people outside of the study likely to be praying for the same patients) and “blinding” (when the putative agent of the effect is an omniscient being). However, assert the authors,
…these are theological questions, and this review proceeds on scientiﬁc principles in that it is a widely held belief that intercessory prayer is beneﬁcial for those who are unwell because God directs the outcome of those for whom prayers are offered differently from those for whom it is not. As noted above, we are not seeking to assess whether God is or is not the agent of action for prayer but, by using the same study designs used to test other interventions in healthcare we will assess the effects of the intervention. For this reason we also exclude from consideration such theological considerations as the injunction “Do not put the Lord your God to the test” (Deuteronomy 6:16) or questions as to whether God generally veils his presence from observation: in the words of the philosopher GF Hegel, “God does not offer himself for observation” (Hegel 2008).
Given their determination to measure “effect not mechanism” and to exclude theological considerations, it seems paradoxical that the authors chose to exclude “distant healing” (DH) studies that “may have included an element of prayer but did not specifically involve personal, focused, committed and organised intercessory prayer on behalf of another alone.” Thus they excluded one of the most famous, purportedly ‘positive’ studies in the field, which had recruited 40 “Healers” with
…an average of 17 years of experience and [who] had previously treated an average of 106 patients at a distance. Practitioners included healers from Christian, Jewish, Buddhist, Native American, and shamanic traditions as well as graduates of secular schools of bioenergetic and meditative healing.
Those ‘healers’ were told “to ‘direct an intention for health and well-being’ to the subject.” Thus, even though there was a religious theme to the choice of ‘healers,’ the imagined ‘mechanism’ of healing was decidedly psychokinetic—it was linear rather than angular, or non-stop rather than 1-stop, if you catch my drift. This was in keeping with the interests of the most important co-author, the late Elisabeth Targ, previously mentioned here.
That’s why the Cochrane authors excluded it and similar DH studies, but c’mon: an influential group of ‘CAM’ enthusiasts, including Targ, Larry Dossey, Victor Sierpina (Distinguished Teaching Professor at the University of Texas Medical School), Mehmet Oz (heh), Marilyn Schlitz (a former member of the NCCAM advisory council), naturopath Leanna Standish (also a former member of the NCCAM advisory council and the Director of Research at the Bastyr University AIDS Research Center), Andrew Weil, Kenneth Pelletier, James Gordon (Chairman of the White House Commission on Complementary and Alternative Medicine Policy), Jeanne Achterberg (who, together with Dossey and Gordon, chaired the “Mind-Body” panel of the 1992 “Workshop on Alternative Medicine,” whose report has debased medicine and medical research for nearly two decades), and many more fairly gush over the potential of ‘nonlocal healing.’ There’s a lotta research money wasted there, so it’s too bad that Cochrane hasn’t offered the same conclusion about the non-stop version of DH that it now has about the layover kind.
I also wonder if the reviewers would have included Targ’s study if that particular exclusion had not held, because Targ was later revealed to have rigged her study to yield “positive” results. She did this after the fact but before the publication, by “data dredging.” I’ve come to expect Cochrane reviewers to remain blissfully ignorant of such departures from polite methodology. Consider their ingenuous response to the Olszewer paper in the chelation review. In this “intercessory prayer” (IP) review are examples that needn’t require the reviewers to venture from the papers themselves. The review characterizes the most famous early ‘positive’ study, Byrd 1988, as double-blinded. That, presumably, follows from this statement in Byrd’s Methods section:
Patients were randomly assigned (using a computer-generated list) either to receive or not to receive intercessory prayer. The patients, the staff and doctors in the [coronary care] unit, and I remained “blinded”, throughout the study. As a precaution against biasing the study, the patients were not contacted again.
Well, OK, but consider this statement in the very next paragraph (emphasis added):
The patients’ first name, diagnosis, and general condition, along with pertinent updates in their condition, were given to the intercessors.
It seems that someone with access to that coronary care unit (CCU) musta not been blinded, and could easily have revealed subject allocation to the subjects themselves and to others. Just sayin’.
The review is ambivalent about the Byrd Score, a composite “severity” score that Byrd devised ostensibly to deal with the problem of multiple outcomes. Here are the results of those outcomes:
Byrd Table 2
Hmmm. The difference that jumps out at you is the incidence of congestive heart failure (CHF). All other differences reported to have achieved statistical significance—use of diuretics, intubation/ventilation, pneumonia, antibiotics, and even cardiopulmonary arrest—likely followed from CHF or from a common antecedent. Since such key outcomes as mortality and duration of CCU and hospital stay were no different between the two groups (surprising given the poor prognosis of CHF, especially 23 years ago), it seems reasonable to discount the CHF difference as either spurious or, as the Cochrane authors correctly acknowledged, due to chance in the context of multiple outcomes.
Not acknowledged by the reviewers were other curious findings in Byrd’s table: if 14 subjects in the control group suffered cardiopulmonary arrest—which involves a blood pressure of approximately zero—how could only 7 subjects in that group have experienced systolic blood pressures below 90? How could only 3 subjects in the IP group have suffered cardiopulmonary arrest—the final common pathway of dying, other than for the special category of ‘brain death’—when more than 4 times that many (13) actually died? Oh yeah, and dead people also have blood pressures below 90, except, apparently, several in each of the groups reported here. I dunno about you, but I’d like to think that any reasonably intelligent physician or scientist would look at that table for a couple of minutes and conclude, “Nope. Nuthin’ goin’ on there.”
The Cochrane reviewers included a study of “retroactive intercessory prayer.” Yup, it means what you’re afraid it means, your double-take notwithstanding. I am not making this up: Check it out. ;-)
All right, you must be thinking, so far I’ve shown you nothing but reasons to be more pessimistic than ever about Cochrane ‘CAM’ Reviews. Next they’ll be declaring that there is not enough evidence either in favour or against the use of exorcisms for demonic possessions, f’crissakes. But remember, the very same reviewers who went for time travel also politely called for a halt in intercessory prayer trials, so something must have swayed them.
Feedback
The answer seems to be found in two Feedback letters. The second is identified only as having been written by “Chris Jackson, anaesthetist.” I don’t know where he or she is from, but I’m guessing he’s what we in the U.S. call an ‘anesthesiologist.’ That’s what I am! Chris, you make me proud. This letter apparently jolted the Cochrane reviewers into noticing that a study they’d included for years, the infamous Cha Intercessory Prayer for IVF study, was, well, infamous enough to finally exclude (in 2009). Jackson also wrote that “RCTs of prayer are meaningless…There’s a lot of pseudoscience being done in this area,” which the reviewers, alas, didn’t buy.
The first Feedback letter is much longer, more adamant and less polite, and—what a kick!—written by other Cochrane reviewers. It begins with condemnation:
This review is riddled by serious flaws such as lack of critical appraisal of the included trials and findings, lack of a necessary discussion of the relevant sources of bias, and undue interference of theological reasoning.
It ends with a call for banishment:
This review does not live up to the scientific standards one can reasonably expect of a Cochrane review. The review as currently published should be withdrawn from the Cochrane Library, not least because it suggests that all scientific studies are meaningless, as we will never know whether one or more gods intervened in our carefully planned experiments.
The authors of this letter are identified as Karsten Juhl Jørgensen, Asbjørn Hrobjartsson and Peter C. Gøtzsche, from the Nordic Cochrane Centre, Rigshospitalet Dept. 3343, Copenhagen, Denmark. The cognoscenti among you might recognize Hrobjartsson and Gøtzsche as the authors of several reviews questioning the ‘power of the placebo,’ a topic that they’ve also reviewed for Cochrane.
I’m happy to report that I needn’t quote any more excerpts from that Feedback letter, even though you can’t read it without paying for the full review, because there’s an even better source. Jørgensen and colleagues turned their letter into a full article that you can read online in the aptly-named Journal of Negative Results in Biomedicine: “Divine Intervention? A Cochrane review on intercessory prayer gone beyond science and reason.” They make several of the points made above and elsewhere in this series (citing Bayes, for example), and many more, because unlike your semi-faithful blogger they were not too impatient to slog through the tedious religious formulations in the Cochrane IP review.
I suspect that it was this article and its associated Feedback letter that led the Cochrane IP reviewers to reverse their previous call for further studies, even if they failed to heed most of the arguments made by Jørgensen et al. Unfortunately, you would only know the last point if you had access to the full Cochrane review, where the exchange is found.
This post is already way too long, so I’ll end by telling you the most amusing example. By now I’m sure you either know or suspect that the “retroactive intercessory prayer” study included in the Cochrane IP review was a joke that the Cochrane reviewers didn’t get. The Danes explained this both in their article and in their Feedback letter, even providing a reference to a subsequent letter by the “retroactive” author in which he pretty much cops to the joke. The Cochrane reviewers, notwithstanding, responded:
Comments made about the Christmas issue of the BMJ and the Leibovici 2001 study in particular are not fully accurate. Several articles in the late December issues of the BMJ are written with humour and some in pure spoof. Most are not. They may be written with humour and have an odd perspective, but are, nevertheless, interesting and well thought out research. The Leibovici 2001 was not in jest. It is a rather serious paper, intended as a challenge.
Yikers.

   Interpreting the Medical Literature
Steven Novella
Link
The science in science-based medicine includes all of science, but relies most heavily on the biomedical literature – published studies that collectively represent our scientific medical knowledge. The scientific basis of medicine is only as good as this body of knowledge and the manner in which it is interpreted and put into practice.
We often discuss on this blog how to evaluate individual studies- the need for blinding, randomization, the importance of study size to meaningful statistical analysis, and other features that distinguish a reliable study from a worthless one. This is important, but only half of the equation. We also at times discuss the medical literature as it relates to a specific medical question or set of related questions – does homeopathy work or are statins beneficial for cholesterol reduction, for example. This requires not only the ability to judge individual studies, but a higher order analysis of the overall pattern of evidence among all relevant studies. Failure to do this, by focusing only on individual studies, results in the failure to see the forest for the trees.
It is this higher order analysis that I wish to discuss in this entry.
The evolution of medical evidence
First, it is important to realize that confident medical judgments or conclusions rarely emerge from single studies – confidence requires a pattern of evidence over many studies. The typical historical course for such evidence is first to begin with clinical observations or plausible hypotheses that stem from established treatments. Based upon this weakest form of evidence preliminary or pilot studies are performed by some interested researchers to see if a new treatment has any potential and is at least relatively safe. If these early studies are encouraging then larger and larger studies, with more tight designs are often performed.
In this early phase of research results are often mixed (some positive, some negative) as researchers explore different ways to use a treatment, different subsets of patients on which to use the treatment, varying doses of medication, or other variables. Outcomes are also variable – should a hard biological outcome be used, or more subjective quality of life outcomes? What about combinations with existing treatments? Is there an additive effect, or is the new treatment an alternative.
It takes time to sort out all the possible variables of even what sound initially like a simple medical question – does treatment A work for condition X. Often different schools of thought emerge, and they battle it out, each touting their own studies and criticizing the others’. This criticism is healthy, and in the best case scenario leads to large, well-designed, multi-center, replicable consensus trials – trials that take into consideration all reasonable concerns where both sides can agree upon the results.
What I just described has often occurred in medical practice, but is also what happens when the system works. There are also many false turns and blind alleys. Sometimes fraud will confuse the literature until it is rooted out by later studies, or a vested interest will skew the studies in one direction, delaying a more definitive and accurate result from emerging from the collective literature. But it is important to realize that even when the process works well, it is very messy, and it takes years to play out.
That is why medicine is an inherently conservative profession – we cannot allow medical practice to blow in the wind of every new study or fad. It takes time to achieve confidence by working through all the messy variables.
But there’s more.
Systematic Reviews vs Meta-Analysis
Perhaps the most useful type of medical paper is the systematic review. These take a great deal of time and work, but when well-done are invaluable. Perhaps the most significant contribution of the Cochrane collaboration – an institution of evidence-based medicine – is to provide standards for systematic reviews and to publish many of them.
A good review will look at all high-quality studies, assess their strengths and weaknesses, and look at the overall pattern of results. For example, such a review may find that there are no studies of sufficient quality to make a recommendation – therefore the evidence is still lacking. Or, it may find that the better a study of a certain treatment the smaller the effect, and the best studies tend to be negative – a pattern consistent with a lack of a real effect (despite the fact that there may be individual positive studies). Such reviews may also point out inconsistencies in the literature indicating the need for further research to resolve the conflicts.
Systematic reviews are not without their pitfalls, however. The reviewers still have to decide which studies to include, and this creates the potential to introduce bias in the results.
Even more concerning – systematic reviews are only as good as the literature it analyzes, and there are systematic problems with the literature itself that I will discuss further below.
A meta-analysis is not a systematic review. Rather, this is the statistical technique of combining multiple studies into one large study by pooling their data. This technique is fraught with multiple sources of bias – which studies to include and how to standardize different populations and outcomes being the two most significant. It is important to realize that a meta-analysis is not new data, but simply a re-analysis of old data with new variables potentially introduced.
A 1997 study published in the NEJM looked at published meta-analyses and compared them to later large well-designed trials and found that they failed to predict the outcome of the later trials 35% of the time. This is not very good, considering that 50% would be random chance. The probability of false positive and false negative were roughly equal in this study.
Biases in the Literature and the work of John Ioannidis
The literature itself has higher order structure – meaning that it is not simply a collection of individual studies. There are patterns in which types of studies tend to get published, and therefore added to the literature. The most basic such pattern is called publication bias, or the file-drawer effect. It refers specifically to the tendency to publish positive studies over negative studies. Researchers are more likely to submit positive studies, and journal editors are more likely to accept them. A recent analysis, for example, also showed that pharmaceutical companies are much more likely to submit studies favorable to their drugs for publication than negative studies. Similarly, it is likely that proponents of any treatment or system are more likely to submit for publication studies that support their specialties or favored modalities.
This has a huge effect on the literature. It affects every systematic review and meta-analysis, which typically will analyze only published studies. Cochrane reviews also encourage looking at unpublished data and paper presentations at meetings, but this is only done in the most complete of reviews. If the collection of published studies is skewed toward the positive, then reviews will likewise be skewed positive.
This is further supported by evidence that initial studies of new treatments tend to be positive and have large treatment effects, and as the literature on a particular question evolves there is a distinct trend toward shrinking effect sizes, with some disappearing completely. Taken at face value, what this means is that the medical literature simply cannot be trusted until we get to the mature phase of large, multi-center, randomized, and well-controlled studies – the most definitive study designs.
There are more subtle problems with the medical literature, as discussed by researcher John Ioannidis. In 2005 he published what has become a seminal paper on why most published studies are false. In the summary he writes:
In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true.
The reason for this is actually simple – science is about discovering new things, and most attempts at guessing what will turn out to be true in science are wrong. If most ideas are wrong, then even with a reasonable study design and threshold for statistical significance, you may get more false positives than true positives – in other words, most published studies will be wrong.
It is worth pointing out that this effect gets worse the more implausible the hypothesis. When studying truly speculative or unlikely treatments, the false positive to true positive ratio increases dramatically.
However, it is also worth pointing out that this does not mean that we cannot trust the medical literature at all – after all, Ioannidis is comparing the medical literature to itself. What this really means is that, assuming later large definitive trials to be correct, earlier smaller trials are likely to be false, and more often than not falsely positive. The assumption here is that we can trust mature, well-designed and large trials.
Recently Ioannidis published another paper, this one more of a thought experiment than analysis of the literature. In it he writes:
This essay makes the underlying assumption that scientific information is an economic commodity, and that scientific journals are a medium for its dissemination and exchange. While this exchange system differs from a conventional market in many senses, including the nature of payments, it shares the goal of transferring the commodity (knowledge) from its producers (scientists) to its consumers (other scientists, administrators, physicians, patients, and funding agencies).
This is an interesting idea – thinking of the medical literature in economic terms. I think Ioannidis makes a convincing case that there is some insight to be gained from this approach. He concludes that positive studies are much more valuable as a commodity than negative studies and therefore this should contribute significantly to the bias in the literature toward not only positive studies, but ones that make bold claims – the very claims that are most likely to later turn out to be incorrect. What this adds up to is yet another reason to be highly suspicious of early studies or bold new claims.
Of course, this analysis is only as good as the stated underlying assumption – is scientific information an economic commodity. One aspect of publication that Ioannidis did not address is peer-review. Journals submit papers to review by experts, who have their own reputations to consider and may not gain from someone else publishing a hot new study. High end journals use this process, and their editorial discretion, ostensibly to filter out bad studies. Yes, they want positive, high-impact publications – but their long term reputation also depends upon these studies not being routinely overturned. This is analogous to the reputation of a judge who is consistently overturned on appeal.
How these competing interests (the desire for exciting press releases while wanting published papers to hold up over time) play out is likely different for each journal, depending upon the editorial competence and temperament. I do hold out, perhaps naively, that the best medical journals value peer-review and reliability over headlines.
Conclusion
All of these trends and patterns in the published medical literature tend to point toward the same conclusion - early on in the research of any new idea in medicine, published studies are highly unreliable and have a huge bias towards being positive and exaggerating effect sizes. However, over time the research often works itself out, and highly reliable studies can be achieved (although no question in science is ever closed).
The work of Ioannidis and others also reinforces one of the core premises of SBM – that plausibility is an extremely valuable quality to take into consideration when evaluating any new idea or treatment. The less plausible a treatment, the less reliable early, small, or poorly controlled studies are likely to be, and the greater the bias towards publishing positive outcomes.
There is also a bright spot in all of this – the literature is not hopeless, and all of the problems discussed above have solutions. We need greater transparency in medical research, for example. This could be achieved by having a central registry of all clinical studies involving human subjects that requires the publication online of the study results. This would make all the data (not a biased subset of it) available to researchers for systematic reviews.
Journal editors need to publish more negative studies, and they need to be especially skeptical of new bold claims. Plausibility needs to be factored more prominently into evidence-based medicine – which is precisely what we mean by science-based medicine. More resources should be allocated to high quality systematic reviews.
The medical literature is in many ways our most important tool and resource. Understanding it at its highest level of organization is crucial to reaching reliable medical conclusions that can be applied to practice – which of course is the ultimate purpose of medical research. Further, it is worth our time and effort to continue to study the literature at this top level with the goal of improving the quality and effectiveness of this critical tool.

   The Importance and Limitations of Peer-Review
Steven Novella
Link
Peer-review is a critical part of the functioning of the scientific community, of quality control, and the self corrective nature of science. But it is no panacea. It is helpful to understand what it is, and what it isn’t, its uses and abuses.
Overview
When the statement is made that research is “peer-reviewed” this is usually meant to refer to the fact that it has been published in a peer-reviewed journal. Different scientific disciplines have different mechanisms for determining which journals are legitimately peer-reviewed. In medicine the National Library of Medicine (NLM) has rules for peer-review and they decide on a case by case basis which journals get their stamp of approval. Such journals are then listed as peer-reviewed.
The basic criterion is that there is a formalized process of peer-review prior to publication – so this presents a barrier to publication that acts as a quality control filter. Typically, the journal editor will give a submitted paper to a small number of qualified peers – recognized experts in the relevant field. The reviewers will then submit detailed criticism of the paper along with a recommendation to reject, accept with major revisions, accept with minor revisions, or accept as is. It is rare to get an acceptance as is on the first round.
The editor also reviews the paper, and may break a tie among the reviewers or add their own comments. The process, although at times painful, is quite useful in not only checking the quality of submitted work, but improving the quality. A reviewer, for example, may point out prior research the authors did not comment on, or may point our errors in the paper which can be fixed.
It is typical for authors to submit a paper to a prestigious journal first, and then if they get rejected to work their way down the food chain until they find a journal that will accept it. This does not always mean that the paper was of poorer quality – the most prestigious journals have tons of submissions and can pick and choose the most relevant or important studies. But sometimes it does mean the paper is mediocre or even poor.
The limitations of Peer-Review
It is important to realize that not all peer-reviewed journals are created equal. Small or obscure journals may follow the rules and gain recognized peer-reviewed status, but be desperate for submissions and have a low bar for acceptance. They also have a harder time getting world-class experts to review their submissions, and have to find reviewers that are also farther down the food chain. The bottom line is that when a study is touted as “peer-reviewed” you have to consider where it was reviewed and published.
Even at the best journals, the process is only as good as the editors and reviewers, who are people who make mistakes. A busy reviewer may give a cursory read through a paper that superficially looks good, but miss subtle mistakes. Or they may not take the time to chase down every reference, or check all the statistics. The process generally works, and is certainly better than having no quality control filter, but it is also no guarantee of correctness, or even the avoidance of mistakes.
Peer-reviewers also have biases. They may be prejudiced against studies that contradict their own research or their preferred beliefs. They may therefore bias the published studies in their favored direction, and may be loath to give a pass to a submission that would directly contradict something they have published. For this reasons editors often allow authors to request or recommend reviewers, or to request that certain people not be asked to be reviewers. Each journal has their own policy. Sometimes an editor will specifically use a reviewer that the authors request not be used, thinking they may be trying to avoid legitimate criticism.
The process can be quite messy, and full of politics. But in the end it more or less works. If an author thinks they were treated unfairly by one journal, they can always go to another or they can talk directly to the editor to appeal a decision and try to make their case.
Perhaps the biggest weakness of peer-review, however, is when an entire discipline of peers is lacking in some fundamental way. For example, there are now many journals dedicated to so-called “alternative medicine” (CAM). The editors of such journals tend to have a pro-CAM bias, and they find reviewers with a pro-CAM bias. So pretty much any pro-CAM article can get published. Some have enough ideological friends at the NLM that they can get approved as peer-reviewed, despite glaring biases in their editorial policy.
Post Publication Peer Review
The term peer-review is sometimes used to refer to the fact that papers are read and reviewed by the broader scientific community once they are published. However, this post-production review should not be confused with “peer-reviewed” and that term should not be used to refer to post-publication review, to avoid confusion.
The process, however, is even more critical to quality control in science. Now, instead of one editor and 2-3 reviewers looking at a study, dozens or hundreds (maybe even thousands) of scientists can pick over a study, dissect the statistics and the claims, bring to bear knowledge from related areas or other research, and provide detailed criticism. This is the real “meat grinder” of science. Hundreds of reviewers are more likely to find problems than the few pre-publication reviewers. Arguments can be tested in the unforgiving arena of the scientific community, weeding out bad arguments, honing others, so that only the best survive.
Conclusion
Here is the bottom line – peer-review is a necessary component of quality control in science, but is no guarantee of quality, and you have to know the details of the journal that is providing the peer-review.

The Power of Replication – Bems Psi Research
Steven Novella on August 29, 2012
Link
I love reading quotes by the likes of Karl Popper in the scientific literature. A recent replication of Bem’s infamous psi research, Feeling the Future, gives us this quote:
Popper (1959/2002) defined a scientifically true effect as that “which can be regularly reproduced by anyone who carries out the appropriate experiment in the way prescribed.”
The paper is the latest replication of Daryl Bem’s 2011 series of 9 experiments in which he claimed consistent evidence for a precognitive effect, or the ability of future events to influence the present. The studies were published in The Journal of Personality and Social Psychology, a prestigious psychology journal. All of the studies followed a similar format, reversing the usually direction of standard psychology experiments to determine if future events can affect past performance.
In the 9th study, for example, subjects were given a list of words in sequence on a computer screen. They were then asked to recall as many of the words as possible. Following that they were given two practice sessions with half of the word chosen by the computer at random. The results were then analyzed to see if practicing the words improved the subject’s recall for those words in the past. Bem found that they did, with the largest effect size of any of the 9 studies.
Needless to say, these results were met with widespread skepticism. There are a number of ways to assess an experiment to determine if the results are reliable. You can examine the methods and the data themselves to see if there are any mistakes. You can also replicate the experiment to see if you get the same results.
Replicating Bem
Bem’s studies have not fared well in replication. Earlier this year Ritchie, Wiseman, and French published three independent replications of Bem’s 9th study, all negative. Of further interest is that the journal that originally published Bem’s article had declined to publish Ritchie et al’s paper claiming that they don’t publish replications. This decision (and editorial policy) was widely criticized, as it reflects an undervaluing of replications.
It’s good to see that the journal has relented and agreed to publish a replication. Galak, LeBoeuf, Nelson, and Simmons should be commended, not only on their rigorous replication but their excellent article, which hits all the key points of this entire episode.
The researchers replicated experiments 8 and 9 of Bem (they chose these protocols because they were the most objective). They conducted 7 precise replications involving a total of 3,289 subjects (Bem’s studies involved 950 subjects). Six of the seven studies, when analyzed independently, were negative, while the last was slightly statistically significant. However, when the data are taken together, they are dead negative. The authors concluded that their experiments found no evidence for psi.
Lessons from Bem
A Bayesian Analysis
At Science-Based Medicine we have advocated taking a more Bayesian approach to scientific data. This involves considering a claim for plausibility and prior probability. Bem and others are fairly dismissive of plausibility arguments and feel that scientists should be open to whatever the evidence states. If we dismiss results because we have already decided the phenomenon is not real, then how will we ever discover new phenomena?
On the other hand, it seems like folly to ignore the results of all prior research and act as if we have no prior knowledge. There is a workable compromise – be open to new phenomena, but put any research results into the context of existing knowledge. What this means is that we make the bar for rigorous evidence proportional to the implausibility of the phenomenon being studied. (Extraordinary claims require extraordinary evidence.)
One specific manifestation of this issue is the nature of the statistical analysis of research outcomes. Some researchers propose that we use a Bayesian analysis of data, which in essence puts the new research data into the context of prior research. A Bayesian approach essentially asks – how much does this new data affect the prior probability that an effect is real?
Wagenmakers et al reanalyzed Bem’s data using a Bayesian analysis and concluded that the data is not sufficient to reject the null hypothesis. They further claim that the currently in vogue P-value analysis tends to overcall positive results. In reply Bem claims that Wagenmakers used a ridiculously low prior probability in his analysis. In reality, it doesn’t matter what you think the prior probability is, the Bayesian analysis showed that Bem’s data has very little effect on the probability that retrocausal cognition is real.
Galak et al in the new study also perform a Bayesian analysis of their own data and conclude that this analysis strongly favors the null hypothesis. They also specifically cite Wagenmakers’ support for the Bayesian approach.
Criticisms of Bem
It’s possible for a study to “look good on paper” – meaning that the details that get reported in the published paper may make the study look more rigorous than it actually was. There may also be alternate ways to analyze the data that give a different picture Ritchie, Wiseman, and French outline several criticisms of Bem’s methods. They mention the Bayesian issue, but also that an analysis of the data shows an inverse relationship between effect size and subject number. In other words, the fewer the number of subjects the greater the effect size. This could imply a process called optional stopping.
This is potentially very problematic. Related to this is the admission by Bem, according to the article, that he peeked at the data as it was coming in. The reason peeking is frowned upon is precisely because it can result in things like optional stopping, which is stopping the collection of data in an experiment because the results so far are looking positive. This is a subtle way of cherry picking positive data. It is preferred that a predetermined stopping point is chosen to prevent this sort of subtle manipulation of data.
Another issue raised was the use of multiple analysis. Researchers can collect lots of data, by looking at many variables, and then making many comparisons among those variables. Sometimes they only publish the positive correlations, and may or may not disclose that they even looked at other comparisons. Sometimes they publish all the data, but the statistical analysis treats each comparison independently. In short what this means is that if you look at 20 comparisons with a 1/20 chance of reaching statistical significance, by chance one comparison would be significant. You can then declare a real effect. But what should happen is that the statistical analysis is adjusted to account for the fact that 20 different comparisons were made, which can potentially make the positive results negative.
Finally there was a serious issue raised with how the data was handled. Subjects occasionally made spelling error when listing the words they recall. The result may have been a non-word (like ctt for cat) or another word (like car for cat). Researchers had to go through and correct these misspellings manually.
The authors point out that these corrections were done in a non-blinded fashion, creating the opportunity to fudge the data toward the positive by how correction choices are made. Bem countered that even if you removed the corrected words the results would still be positive, but that is still methodologically sloppy and is likely still relevant, for reasons I will now get into.
Researcher Degrees of Freedom
As we see, there were many problems with the methods and statistical analysis of Bem’s original paper. Bem argues that each problem was small and by itself would not have changed the results. This argument, however, misses a critical point, made very clear in another recent paper – one that was also cited and discussed in the Galak paper.
Simmons et al published a paper demonstrating how easy it is to achieve false positive results by exploiting (consciously or unconsciously) so-called “researcher degrees of freedom.” In the abstract they write:
“In this article, we accomplish two things. First, we show that despite empirical psychologists’ nominal endorsement of a low rate of false-positive findings (! .05), flexibility in data collection, analysis, and reporting dramatically increases actual false-positive rates. In many cases, a researcher is more likely to falsely find evidence that an effect exists than to correctly find evidence that it does not. We present computer simulations and a pair of actual experiments that demonstrate how unacceptably easy it is to accumulate (and report) statistically significant evidence for a false hypothesis.”
In my opinion this is a seminal paper that deserves wide distribution and discussion among skeptics, scientists, and the public. The paper discusses the fact that researchers make many decisions when designing and executing a study, and analyzing and reporting the data. Each individual decision may have only a small effect on the final outcome. Each decision may be made perfectly innocently, and can be reasonably justified.
However, the cumulative effect of these decisions (degrees of freedom) could be to systematically bias the results of a study toward the positive. The power of this effect is potentially huge, and likely results in a significant bias towards positive studies in the published literature.
But even worse, this effect can also be invisible. As the authors point out – each individual decision can seem quite reasonable by itself. The final published paper may not reflect the fact that the researchers, for example, looked at three different statistical methods of analysis before choosing the one that gave the best results.
The authors lay out some fixes for this problem, such as researchers disclosing their methods prior to collecting data (and no peeking). But another check on this kind of bias in research is replication.
The Power of Replication
The degrees of freedom issue is one big reason that replicating studies, especially precise replications, is so important. A precise replication should have no degrees of freedom, because all the choices were already made by the original study. If the effect being researched is real, then the results should still come out positive. If they were the result of exploiting the degrees of freedom, then they should vanish.
There are also the other recognized benefits of replication. The most obvious is that any unrecognized quirky aspects of study execution or researcher biases should average out over multiple replications. For this reason it is critical for replications to be truly independent.
Another often missed reason why replications are important is simply to look at a fresh set of data. It is possible for a researcher, for example, to notice a trend in data that generates a hypothesis. That trend may have been entirely due to random clustering, however. If the data in which the trend was initially observed is used in a study, then the original random clustering can be carried forward, creating the false impression that the hypothesis is confirmed.
Replication involves gathering an entirely new data set, so any prior random patterns would not carry forward. Only if there is a real effect should the new data reflect the same pattern.
Proper Skepticism
Prominently displayed at the top of the Society for Psychical Research’s website is this quote:
“I shall not commit the fashionable stupidity of regarding everything I cannot explain as a fraud.” – C.G.Jung
Clearly that quote reflects the prevailing attitude among psi researchers of external skepticism of their claims and research. Every skeptic who has voiced their opinion has likely been met with accusations of being dismissive and closed-minded.
But this is a straw man. Skeptics are open to new discoveries, even paradigm-changing revolutionary ideas. Often I am asked specifically – what would it take to make me accept psi claims. I have given a very specific answer – one that applies to any extraordinary claim within medicine. It would take research simultaneously displaying the following characteristics:
1 – Solid methodology (proper blinding, fresh data set, clearly defined end points, etc.)
2 – Statistically significant results
3 – Absolute magnitude of the effect size that is greater than noise level (a sufficient signal to noise ratio)
4 – Consistent results with independent replication.
Most importantly, it would need to display all four of these characteristics simultaneously. Psi research, like most research into alternative medicine modalities like homeopathy and acupuncture, cannot do that, and that is why I remain skeptical. These are the same criteria that I apply to any claim in science.
In addition, I do think that prior probability should play a role – not in accepting or rejecting any claim a priori, but in setting the threshold for the amount and quality of evidence that will be convincing. This is reasonable – it would take more evidence to convince me that someone hit Bigfoot with their car than that they hit a deer with their car. There is a word for someone who accepts the former claim with a low threshold of evidence.
You can convince me that psi phenomena are real, but it would take evidence that is at least as solid as the evidence that implies that such phenomena are probably not possible.
It is also important to recognize that the evidence for psi is so weak and of a nature that it is reasonable to conclude it is not real even without considering plausibility. But it is probably not a coincidence that we consistently see either poor quality or negative research in areas that do have very low plausibility.
Conclusion
The least important implication of the recent paper by Galak et al is that it provides further evidence against psi as a real phenomenon, and specifically against the claims of Daryl Bem. Psi is a highly implausible hypothesis that has already been sufficiently refuted, in my opinion, by prior research.
The paper is perhaps a milestone, however, in other important respects:
– It is an admission of sorts by The Journal of Personality and Social Psychology of the importance of precise replication and a reversal of their prior decision not to publish such research.
– The paper highlights the role, and possible superiority, of Bayesian analysis as a method for looking at experimental data.
– It highlights the role of researcher degrees of freedom in generating false positive data, and replication as one solution to this problem.
– This large, rigorous, and negative replication establishes that studies published in peer-reviewed journals with positive and solid-appearing results can still be entirely wrong. It therefore justifies initial skepticism toward any such data, especially when extraordinary claims are involved.
The insights provided by this excellent paper reflect many of the points we have been making at SBM, and should be applied broadly and vigorously to alternative medicine claims.

Is there a reproducibility “crisis” in biomedical science? No, but there is a reproducibility problem
Reproducibility is the key to scientific advancement. It has been claimed that we suffer from a “reproducibility crisis,” but in reality it is a chronic problem in reproducibility. Here we will look at the scope of the problem and strategies to address it.
David Gorski on June 6, 2016
Link
NOTE: Anyone who has seen several derogatory articles about me on the web and is curious about what the real story is, please read this and this.
Most scientists I know get a chuckle out of the Journal of Irreproducible Results (JIR), a humor journal that often parodies scientific papers. Back in the day, we used to chuckle at articles like “Any Eye for an Eye for an Arm and a Leg: Applied Dysfunctional Measurement” and “A Double Blind Efficacy Trial of Placebos, Extra Strength Placebos and Generic Placebos.” Unfortunately, these days, reporting on science is giving the impression that the JIR is a little too close to the truth, at least when it comes to reproduciblity, so much so that the issue even has its own name and Wikipedia entry: Replication (or reproducibility) crisis. It’s a topic I had been meaning to write about again for a while. Fortunately, A recent survey published in Nature under the somewhat clickbaity title “1,500 scientists lift the lid on reproducibility” finally prodded me to look into this question again. Before I get to the survey itself, though, I can’t help but do my usual pontificating to provide a bit of background.
The spectrum of reproducibility and the inherent messiness of science
Having been a PhD-holding scientist now for well over 20 years, and a physician since 1988, I’ve done a lot of experiments, published a fair number of papers in the peer-reviewed biomedical literature (but significantly fewer than I wish I had, because otherwise I would have reached the rank of Professor years ago instead of this year), and grappled with the problem of reproducibility in my areas of research. One of the first things I learned in graduate school is that, as nice and neat as the science sounds when it’s taught in the classroom, it’s anything but nice and neat. Indeed, when it comes to some molecular biology techniques, we used to joke about sacrificing goats to appease the gods of molecular biology to get experiments to work. Then, of course, the more complicated the experiment, the more ways there are for it to go awry. Of course, there is a difference between a technique, such as Western blot or PCR not working, and an experiment giving a result that can’t be reproduced. Both, however, happen, and it can be devilishly difficult to track down the cause. Indeed, I like to point out that one of the most difficult aspects of science to convey to the general public, particularly about science-based medicine is just how messy it can be. One of the first lessons graduate students learn as they embark on their doctoral research is that early reports in the peer-reviewed literature are by their very nature tentative and have a high probability of ultimately being found to be incorrect—or, more often, only partially correct.
Unfortunately, that is not science as it is imbibed by the public. Fed by too-trite tales of simple linear progressions from observation to theory to observation to better theory taught in school, as well as media portrayals of scientists as finding answers fast, most people seem to think that science is able to generate results virtually on demand. This sort of impression is fed even by shows that I used to watch a few years ago—and even—liked—for their ability to excite people about science, for instance CSI: Crime Scene Investigation and its offspring. In their heyday, these shows portrayed beautiful people wearing beautiful pristine lab coats back lit in beautiful labs using perfectly styled multicolored Eppendorf tubes doing various assays and getting answers in minutes that normally take hours, days, or sometimes weeks. Often these assays are all done over a backing soundtrack consisting of classic rock or newer (but still relatively safe) “alternative” rock. And that’s just for applied science, in which no new ground is broken and no new discoveries made. And don’t even get me started on the stereotypical medical examiners in so many crime show dramas who can generate DNA results or detailed chemical analyses seemingly almost instantly. Real scientists know that life (and science) are complicated, much more than they are on television and in other works of fiction.
Still, the messy nature of scientific research doesn’t automatically mean that reproducibility isn’t a problem. At the very least, we as scientists should do all that we can to minimize the difficulty reproducing experiments that we do. One major reason, aside from wanting to get our results right, is that the first step in building on any scientific finding is to reproduce the experiments that led to that finding before going on to do more experiments to expand on or more deeply investigate that finding. When experiments can’t be reproduced, other labs waste a lot of time, effort, and resources. On the other hand, some experimental results are wrong, and there will always be experimental results that turn out to be wrong. Attempts to reproduce those results are how scientists find out they’re wrong. The question is: How much irreproducible science is unavoidable? Is there really a “replication crisis” or “reproducibility crisis” or whatever you want to call it? And if there is such a crisis, what should we scientists do to address it?
As a scientist quoted in the Nature article about the survey notes:
Failing to reproduce results is a rite of passage, says Marcus Munafo, a biological psychologist at the University of Bristol, UK, who has a long-standing interest in scientific reproducibility. When he was a student, he says, “I tried to replicate what looked simple from the literature, and wasn’t able to. Then I had a crisis of confidence, and then I learned that my experience wasn’t uncommon.”
Indeed. I myself went through just such a rite of passage.
A personal anecdote on reproducing scientific results
One of the key results trumpeted by this survey (more details on the survey and its findings, strengths, and shortcomings in a moment) is that more than 70% of researchers have tried and failed to replicate another scientist’s experiments. If anything, I suspect that number is probably low. In any case, I can certainly say that I’m one of those researchers who’s tried and failed to replicate another researcher’s results. No, maybe that’s not quite correct, as you will see.
Back in 1996, as part of my surgical oncology fellowship at the University of Chicago, I did research in the laboratory of the chair of the Department of Radiation and Cellular Oncology. It was there that I first learned of the work of one of my all-time most admired scientists, surgeon-scientist Dr. Judah Folkman, who was basically the father of modern tumor angiogenesis research. Angiogenesis is the normal physiologic process of growing new blood vessels. This process is critical to many normal bodily functions, such as wound healing, the menstrual cycle, and others, but tumors hijack the process to supply themselves with blood and the nutrients it brings. Blocking angiogenesis, Folkman hypothesized, could therefore be an effective anticancer strategy. In any case, I admired Folkman so much that one of my earliest posts for Science-Based Medicine was a tribute to him after his sudden death from a heart attack at age 74. The way we first became acquainted was through our laboratory’s collaboration with him to study the effect of combining two of his angiogenesis inhibitors, angiostatin and endostatin, with radiation therapy in rodent models of cancer.
Dr. Folkman’s discovery of angiostatin and endostatin began with an ingenious strategy that began from the clinical observation that sometimes tumor metastases appear shortly after the operation to remove the primary tumor. Folkman found a mouse tumor model that mimicked this behavior and in the early 1990s did a series of pioneering experiments. With a strain Lewis lung carcinoma cells of low metastatic potential (LLC-LM), when cells are injected into C57BL/6 mice and allowed to grow subcutaneously, if the tumor is left alone, mice develop only microscopic lung metastases. These metastases do not grow and kill the mouse. If, however, the primary cancer is removed, then many large lung metastases grow rapidly. The results of the experiment above strongly implied that the primary tumor was secreting something that suppresses the growth of microscopic metastases. After this, the Folkman group did what we like to call “brute force” science, collecting mouse urine and analyzing it for tumor suppressive activity until they were able to purify a single 38 kDa peptide, which they designated angiostatin. This involved analyzing literally gallons of mouse urine. (Who said science isn’t glamorous?) Once Folkman’s group had a bunch of angiostatin on hand, it performed the following experiment. Two groups of mice were injected with LLC-LM and the tumors allowed to grow to a certain size, after which they were surgically removed. One group was treated with angiostatin, and the control group with saline. The result was that the control group developed massive lung metastases and died, while the group treated with angiostatin had microsocopic lung metastases that never grew beyond a ball of cells. Dr. Folkman then demonstrated that it was the inhibition of angiogenesis by the angiostatin that kept these tumors in check. Ultimately, he used a similar method to discover endostatin, and later he demonstrated that endostatin could induce tumor dormancy in mice.
You can see why Ralph Weichselbaum, my research mentor, wanted to test combining angiostatin with radiation therapy. Our results were ultimately published in Nature, the only Nature paper on my CV (and, alas, not even as first author). Science works, right? Yes, it does, but the path to these results was not straight. It was widely known at the time that other laboratories were having difficulty reproducing Folkman’s striking results. In our case, we were not observing nearly as potent an antitumor effect as Folkman had described with angiostatin. We wondered if it was something to do with the angiostatin itself, which was being made in bacteria from a plasmid by our collaborators at Northwestern University. Given that Folkman was one of the best scientists I ever met (and I did have the opportunity to meet him on three occasions), none of us doubted his results and assumed that it must be something we were doing.
So what happened next? Weichselbaum contacted Folkman, who provided reagents, protocols, and advice, as well as some angiostatin made in his laboratory. It turns out that the peptide we were making was easily denatured (unfolded), which was why it was not as potent as Folkman had reported. Now here’s why I say we couldn’t replicate his results. It’s because we couldn’t fully replicate his results. Our angiostatin inhibited the growth of a wide variety of tumors, but, even after applying the tweaks to our angiostatin production suggested by Folkman, in our hands angiostatin never inhibited tumor growth as potently as Folkman had reported. So in other words, there could easily have been something else going on that we never figured out. Be that as it may, Folkman had the best attitude I’ve ever seen in a scientist regarding reproducibility, as we learned later when we heard of how he had done the same thing for several other labs, even to the point of dispatching one of his postdocs to help other investigators to get angiostatin and endostatin to work. Still, few investigators could ever quite replicate Folkman’s initial results, although many demonstrated that angiostatin and endostatin were potent angiogenesis inhibitors.
Eventually, angiogenesis inhibitors were clinically validated, in particular Avastin, which is simply a humanized monoclonal antibody against vascular endothelial growth factor (VEGF). (I also played with anti-VEGF antibodies back in the day.) Unfortunately, no angiogenesis inhibitor in humans has ever been as potent as angiostatin and endostatin were in mice. Angiogenesis inhibitors were a useful addition to our anticancer armamentarium, but, contrary to how they were portrayed in 1998, they were no magic bullet.
The point of this anecdote is that reproducibility falls a spectrum. Did I fail to reproduce Folkman’s results? Yes, I reproduced the general result that angiostatin inhibited tumor growth by blocking angiogenesis, but, no, I didn’t reproduce an effect anywhere near as powerful as the one Folkman had reported. Replication of a result can range from total failure to replicate (e.g., I had failed to show any antitumor effect of angiostatin at all) to partial failure to replicated (e.g., what actually happened) to success at replication (e.g., I had shown angiostatin to block tumor growth as powerfully as Folkman had).
Survey says: Reproducibility is a crisis!
Now let’s take a look at the Nature survey. It’s not a scientific survey or even a poll, really, which made me think of dismissing it almost out of hand. Basically, Nature e-mailed the survey to its readers and advertised it on affiliated websites and social media outlets as being “about reproducibility.” So, in other words, this is nothing even resembling a sampling designed to mirror the scientific community, as political polls are designed to mirror the population being polled. Nature itself even blithely notes that the survey “probably selected for respondents who are more receptive to and aware of concerns about reproducibility.” (“Probably”?) Even so, given that it’s basically an Internet poll, I don’t think the survey is without merit, as it does suggest that there is at least a widespread perception among scientists that there is a problem.
For example:
More than 70% of researchers have tried and failed to reproduce another scientist’s experiments, and more than half have failed to reproduce their own experiments. Those are some of the telling figures that emerged from Nature’s survey of 1,576 researchers who took a brief online questionnaire on reproducibility in research.
The data reveal sometimes-contradictory attitudes towards reproducibility. Although 52% of those surveyed agree that there is a significant ‘crisis’ of reproducibility, less than 31% think that failure to reproduce published results means that the result is probably wrong, and most say that they still trust the published literature.
The average lay person might find it odd that half of the respondents had failed to reproduce their own research, but it makes more sense if you look at the actual survey questions, to which the answers were Yes, No, or “I can’t remember”:
    • Tried and failed to reproduce one of your own experiments
    • Tried and failed to reproduce someone else’s experiment
    • Published a successful attempt to reproduce someone else’s work
    • Published a failed attempt to reproduce someone else’s work
    • Tried and failed to publish a successful reproduction
    • Tried and failed to publish an unsuccessful reproduction
I’ve failed to reproduce my own experiments before. When that happens, I either figure out why I couldn’t reproduce the results of that experiment, or I don’t publish and move on to something else. (Of course, that ignores the question of how hard and long I keep plugging away to find out why I can’t reproduce, a question impacted by multiple factors.) That’s how science works, and spurious results are not uncommon. It’s why we often do experiments in triplicate and repeat the same experiment.
One interesting result of this survey is which specialties view their published results as being the most reliable. Of note, physicists and chemists have the most confidence in their published literature, with medicine being among the lowest. I can think of potential reasons for that. Physicists, for instance, can often do many more replicates of their measurements than is possible in medicine; indeed, in medicine, given ethics of clinical trials, there are often just a handful of trials addressing major questions—sometimes only one.
Is reproducibility in science a crisis?
In a word, no.
I don’t like the word crisis to describe what are in fact problems. The word crisis implies an acute time of difficulty or danger, a turning point, or a time when a difficult decision must be made. Reproducibility in science is a problem, a chronic problem, in fact. It is not a crisis, and there is nothing in this survey that suggests we’re coming to a turning point or something horrible is about to happen if we don’t act. In fact, I’m not even convinced that the problem is quite as severe as it is being portrayed. For example, one of the studies frequently cited as evidence that only 10% of biomedical science can be reproduced. As I discussed when I analyzed that claim when it was first made, that survey was written by Lee M. Ellis and a former Amgen executive (C. Glenn Begley) and looked at whether pharmaceutical company scientists could reproduce preclinical results from 53 “landmark” studies as they searched for a way to target new molecular mechanisms those results revealed. As I pointed out at the time, preclinical research is, by definition, preclinical. It’s the groundwork, the preliminary research, that needs to be done to determine the plausibility and feasibility of a new treatment before testing it out in humans. As such, preclinical research encompasses basic research and translational research and can include biochemical, cell culture, and animal experiments.
It’s worth reiterating that what was being discussed was, essentially, frontier science published in very high impact journals, which is why it struck me at the time as rather strange that the authors found it so amazing and deplorable that much of the science at the very frontiers turns out not to be correct when tested further. As I’ve discussed on multiple occasions, the science that is published in the highest profile, most prestigious journals is almost by definition the most tentative science. Given that, it is surprising how much of what is published in such journals actually does stand the test of time, but it should not be surprising that much of it does not. However, the very prestige of such journals gives such research seemingly more authority than research published in less prestigious journals. Moreover, the Amgen executive who co-authored this report led a group that scoured high impact journals for cutting edge studies that appeared to have identified promising molecular targets. Then he had a veritable army of scientists, about 100 of them in the Amgen replication team according to this news report, who were ready to pounce on any published study that suggested a molecular target the company deemed promising. No wonder he could replicated only 11% of the results, particularly given that their definition of “non-reproduced” was assigned “on the basis of findings not being sufficiently robust to drive a drug-development programme.” So in reality, the failure of reproducibility in this oft-cited article is a failure of being able to extend the results sufficiently to justify the resources needed to translate the result to humans, which is a very different thing than a failure to reproduce the experimental results.
Lost in the discussion of the reproducibility problem are a lot of these nuances, but, again, that doesn’t mean there isn’t a problem, and it doesn’t mean that scientists shouldn’t do something, but what?
Defining the reproducibility problem
One major issue that reformers run head on into whenever discussing reproducibility is that there is no consensus on what, exactly, constitutes adequate reproducibility in science. An accompanying editorial notes this very issue:
What does ‘reproducibility’ mean? Those who study the science of science joke that the definition of reproducibility itself is not reproducible. Reproducibility can occur across different realms: empirical, computational and statistical. Replication can be analytical, direct, systematic or conceptual. Different people use reproducibility to mean repeatability, robustness, reliability and generalizability.
Economists and social scientists often use the term to mean that computer code and data are available so that someone would be able, if so inclined, to redo the same analysis using the same data. For bench scientists, who made up most of our respondents, it usually means that another scientist using the same methods gets similar results and can draw the same conclusions. We asked respondents to use this definition.
Even with a fixed definition, the criteria for reproducibility can vary dramatically between scientists. Senior scientists will not expect each tumour sample they examine under a microscope to look exactly like the images presented in a scientific publication; less experienced scientists might worry that such a result shows lack of reproducibility.
In an article from last year about scientific reproducibility, John Ioannidis and C. Glenn Begley observed in a much better discussion of the problem of reproducibility than the one Begley co-authored that I mentioned above:
There is no clear consensus at to what constitutes a reproducible study. The inherent variability in biological systems means there is no expectation that results will necessarily be precisely replicated. So it is not reasonable to expect that each component of a research report will be replicated in perfect detail. However, it seems completely reasonable that the one or two big ideas or major conclusions that emerge from a scientific report should be validated and with-stand close interrogation.
I guess by that definition, I did replicate Judah Folkman’s results after all!
Addressing the causes of the reproducibility problem
The problem of defining what constitutes adequate reproducibility is not a trivial one, and until there is a consensus, there will only be so much that can be done. Still, the Nature survey also asked scientists what they have done and what should be done to make research more reproducible. It turns out that two-thirds of the respondents had instituted procedures to increase reproducibility, one-third within the last five years.
As for the causes of lack of reproducibility, the scientists surveyed listed the usual suspects, such as selective reporting, “publish or perish” pressure, low statistical power, insufficient replications in the original lab, poor experimental design, variable technical expertise, variability in reagents, and even fraud. To show how subtle these problems can be, it’s useful to relate another anecdote, this time from the survey:
Consolidating methods is a project unto itself, says Laura Shankman, a postdoc studying smooth muscle cells at the University of Virginia, Charlottesville. After several postdocs and graduate students left her lab within a short time, remaining members had trouble getting consistent results in their experiments. The lab decided to take some time off from new questions to repeat published work, and this revealed that lab protocols had gradually diverged. She thinks that the lab saved money overall by getting synchronized instead of troubleshooting failed experiments piecemeal, but that it was a long-term investment.
This is not an uncommon tale. The “institutional” memory of a laboratory is something that is very hard to maintain, given that, other than the principal investigator and (sometimes) a permanent technician and/or lab manager, most personnel in labs are only there for at most a few years to get their PhD or do a postdoctoral fellowship. Turnover is high by design. Often there are little “tricks” or nuances to various experimental techniques to get them to work well that are lost when someone leaves a lab. That’s why maintaining protocol notebooks is so important, but few labs do this as rigorously as they should.
The scientists surveyed identified a number of interventions that could improve reproducibility, including a better understanding of statistics, better mentoring, more robust experimental design, more within-lab validation, more time checking notebooks, journals enforcing standards, and incentives for formal replication. These are similar to proposals surveyed by Ioannidis and Begley last year:
    • Editors solicit replication bids
    • Plea to improve editorial standards
    • Reward quality rather than quantity
    • Emphasis on hypothesis testing research
    • Prospective, rigorous experimental plan
    • Improved understanding of statistics
    • Improved experimental design
    • Systematic reviews of animal studies
    • Use clinically relevant concentrations
    • Consider litter effects
    • Recommendations to improve computational biology
    • Focus on reproducibility in training, grants, journals
    • Pathology: Biospecimen quality control
    • Microarray analyses: Provide data access
    • Psychology: open data, methods and workflow
    • Meta-analyses of animal data
    • Judge academics on quality, reproducibility, sharing
    • Greater institutional responsibility
    • Apply greater skepticism to new technologies
Ioannidis and Begley do note, though:
The fundamental problem with most, if not all, of these proposals is the requirement for investigators, institutions, and journals to willingly comply: it is not at all clear how reasonable recommendations will be implemented or monitored while they remain voluntary. Conversely, were they to be mandatory, then one has to examine carefully how they would be enforced and by whom. The details of how to make these changes work can have a major effect on their efficiency.
Indeed. One can easily envision selective enforcement of such rules that give famous and influential scientists an easier time. Making such a mandatory system fair would be a major challenge.
This, of course, brings us to a major part of the problem, namely incentivization and cost. Scientists have long bemoaned that there is little incentive to publish a positive replication of another scientist’s experiment. That’s why getting “scooped” by another scientist can be so disastrous. Because the result is no longer novel, the scientist who gets scooped will have a hard time publishing his results in the better journals. If that result took a lot of time and resources to obtain, not being first to publish can be devastating to future publication and future funding. Worse, there’s even less incentive to publish negative replications.
In the meantime, as these issues are hashed out in the scientific community, the NIH has acted, instituting new requirements for reproducibility and rigor for research grants and mentored career development award applications. The NIH now requires a discussion of the strengths and weaknesses of previous research and the scientific premise, a description of rigorous experimental design and how bias will be eliminated, consideration of sex and other key variables, and validation of key biological and/or chemical reagents. Regarding the latter, believe it or not, a lot of cancer cell lines out there, when tested, turn out not to be the cell line they were thought to be. As for eliminating bias, again, believe it or not, results from many preclinical experiments are not measured in a blinded fashion, allowing observer bias to taint the results.
Conclusion: Maybe not a crisis, certainly an opportunity
The new NIH rules are a step in the right direction but clearly don’t go far enough. I don’t believe that reproducibility in science is in “crisis,” as so many are claiming, but I do believe it’s a significant problem that needs to be addressed in a thoughtful way. I also have to concede that it’s scientists’ fault that we’re in the mess we’re in and that we haven’t addressed problems with reproducibility more robustly before now, given that this problem has been festering for a while. If it takes labeling the problem as a “crisis” to get some action, I suppose I can live with that.
In considering how to encourage good science and discourage bad science, it is important to note that not all science, particularly biomedical science, should be assumed or expected to result in findings that have direct applications or to result in treatments for humans. As Ioannidis and Begley put it, an efficacy “of 100% and waste of 0% is unlikely to be achievable”, even as they note that there is “probably substantial room for improvement.” It is also important to note that, contrary to the way some paint this problem, the concerns about reproducibility in science don’t invalidate the scientific method itself nor disprove “scientism.” Science-based medicine has yielded incredible benefits to human health over the last 150 years. Indeed, the solutions to this problem being proposed are intended to enhance the rigorous application of science, not to abandon it. Finally, I can’t help but note that it is scientists themselves who are being openly self-critical and debating how to fix perceived problems in science. That is a major strength, not weakness, of science.

Where Are We With the Replication “Crisis”
The replication problem is not as bad as the sensational reporting has suggested. But it is still a legitimate issue that needs to be addressed. 
Steven Novella on February 28, 2018
Link

In 2015 Brian Nosek and 269 co-authors published their attempts at replicating 100 psychological studies from high impact journals. They reported that only 39 of the 100 attempts were successful, which means that 61% of the original results could not be replicated.
These results kicked off what has become known as the “Replication Crisis” in science. Only three years later, however, a more nuanced picture has emerged. As our own David Gorski has also noted – there is more of a problem than a crisis.
I think this reflects the position we often find ourselves taking here at SBM. We are somewhere between the two extremes. At one end are those who are naive or true-believers, who will cite one study as if it establishes a claim. At the other end are those who would dismiss the findings of science as hopelessly flawed.
The reality is somewhere in between. Science is hard and requires rigorous methods and critical questioning. It is also highly flawed. But, the results can be meaningful and progress does grind forward. What we need is a high standard of quality and thoughtful and thorough analysis of results. When applied to medicine, that is SBM in a nutshell.
Background on the Replication Crisis
Replication is the cornerstone of quality control in science, and so failure to replicate studies is definitely a concern. How big a problem is replication, and what can and should be done about it?
As a technical point, there is a difference between the terms “replication” and “reproduction” although I often see the terms used interchangeably (and I probably have myself). Results are said to be reproducible if you analyse the same data again and get the same results. Results are replicable when you repeat the study to obtain fresh data and get the same results.
There are also different kinds of replication. An exact replication, as the name implies, is an effort to exactly repeat the original study in every detail. But scientists acknowledge that “exact” replications are always approximate. There are always going to be slight differences in the materials used and the methodology.
So, what is really meant by “exact” replication is that there are no differences that should reasonably affect the outcome.
There are also conceptual replications, in which different methods are used to look for the same basic phenomenon.
Regardless of where a replication is on the spectrum from exact to conceptual, there are significant technical challenges. Whenever any reagents or substances are used in research, they have to be obtained from a source, and there may be meaningful differences in the quality or exact nature of the material. Whenever animals are used, then it is possible that there are differences in the genetics or health of the animals. There are reports of suppliers providing the wrong animal lines, invalidating any researchers who relied upon them. The same is true of cell lines for basic science research.
In short, there are many things that can technically go wrong. Some research also involved great technical skill, and failure to replicate may simply be a factor of lacking technicians with the proper skill or experience.
Further, papers do not always provide enough detail in the methods section for someone else to follow cookbook style. They should, but they often don’t.
When evaluating the results of replications the outcomes are also not binary (success or failure). Results may still be positive, but with a smaller effect size. This is a common outcome, so common it has a name – the “decline effect.” Or, only part of the results may replicate, or only under certain conditions.
The more the original results replicate, however, the more “robust” they are considered to be. This gets to how much they are likely to be generalizable to the real world.
Research on replications
The 2015 Nosek study caused quite a stir that still resonates in the public consciousness. However, a later reanalysis by Gilbert et al found fatal flaws in Nosek’s methodology. For example, if you separate the replications into high fidelity (studies that accurately reproduced the original study, as judged by the original authors) and low fidelity, Gilbert found that the low fidelity studies had four times the failure rate as the high fidelity studies. When looking at all the data he concluded that the Nosek study actually showed that when you don’t do replications correctly you don’t get the same results, but when you do replications correctly you mostly do get the same results.
Gilbert also found significant sampling error in the choice of studies to replicated. Finally he found that the authors did not properly correct for statistical variation. In other words, if you replicate 100 true positive studies, some of the replications will be negative by chance alone. What you really need to know is – do the studies examined fail to replicate at higher than chance? Gilbert and his coauthors found that when you properly examine Nosek’s evidence, the answer is no. The results were consistent, they conclude, with all 100 original studies being true positives. (You cannot draw that conclusion, but it is consistent with the data, within statistical noise.)
The “replication crisis” bell, however, cannot be unrung. This is not necessarily a bad thing, if it brings attention to the replication “problem” that is real.
How have other replication efforts fared (or – how replicable are replication studies)? One of the more famous replication efforts is The Reproducibility Project: Cancer Biology. David Gorski has been following the results here.
This is an effort to replicate (despite the name) basic science cancer studies to see how replicable they are. The first batch of five studies showed that two replicated, one did not, and the other two could not be interpreted because of technical failures to pull off the replications. So – a 40% success rate, but really 2/3 of the studies that could be completed.
Now a second batch of replication studies has been published:
“Overall, he adds, independent labs have now “reproduced substantial aspects” of the original experiments in four of five replication efforts that have produced clear results.”
That’s good, but notice the caveats. The studies replicated “substantial aspects” of the original experiments, but not necessarily all. Also, this excludes studies that could not produce clear results.
But still, that is pretty good. I think this is consistent with the overall impression shared by David and I – the replication problem is not as bad as the sensational reporting has suggested. But it is still a legitimate issue that needs to be addressed.
The concerns are also not limited to psychology or biomedical research. When surveyed, scientists in every specialty report that they sometimes fail to replicate the work of their colleagues. A recent study attempting to replicate economics research found that 49% of the studies could be replicated, even with help from the original authors.
It should also be noted that most replication assessments focus on high impact journals. The problem is presumably worse if you sample studies from progressively less prestigious scientific journals, but that has not been directly examined.
The way forward
There are some obvious solutions that will at least improve the situation and which have broad support, but will still be challenging to implement, largely because of cultural and institutional inertia.
First, we need to do and publish more replications. Some researchers have suggested that internal replications should simply be built into research methodology.
In other words – a research lab should follow up their preliminary results with an exact replication of their own before attempting to publish. This would weed out many false positives before they ever contaminate the literature. Some labs already do this, but it should become standard, which means that journals should require it.
There should also be more academic focus on exact replications, meaning that researchers should be encouraged to do such research, and be given grants, credit, and promotions based on such research. Instead there is an acknowledged focus on new and sexy research (the exact kind that is least likely to be replicable).
It is also interesting to think about a specialty in experimental replications. In other words, imagine a lab of researchers with high technical skill, and the knowledge and expertise necessary to execute extremely rigorous research. Their primary focus could be identifying important published results, then doing exact replications to see if the results are real and robust. Such labs might be a “rite of passage” for any research results before they are considered reliable.
This kind of thing already exists in some contexts. For example, pharmaceutical companies have labs (or contract with private labs) whose job is to replicate academic studies to see if they are real before they invest on trying to develop a new drug based on that research. The financial stakes are high for the company, so it is cost effective for them to invest in exact replications.
There are, of course, downsides to all such recommendations. Doing all these replications takes resources (there is a cost to everything). Further, doing an expert replication requires not only generic expertise in research methodology, but specific expertise in the often narrow topic of interest. As I stated above, sometimes research requires specific technical skills and knowledge, and only researchers dedicated to a narrow area of research might have those skills.
In the end there is no perfect solution. But what we want to consider is – where is the sweet spot of trade-offs to maximize the return on our research investment? It seems that now the balance is shifted toward false positives and innovation, with insufficient priority given to replication and confirmation.
Perhaps we just need to tweak this balance to get more in the sweet spot and optimize scientific advance. It’s a great conversation to have, and represents the true strength of science. No other human endeavor spends so much time wringing its hands about its own methodology and validity.

Reproducibility Follow Up
Let’s explore dueling narratives about the reproducibility “crisis.”
Steven NovellaonOctober 16, 2019
Link


In 2016 Nature published a survey of scientists regarding their experiences with reproducibility. They found that 38% reported they felt there was a slight problem with reproducibility in their field, while 52% reported a significant problem. In medicine respondents felt, on average, that about half of the published literature was reliable. This survey helped fuel the narrative that there is a reproducibility “crisis” in science. They go on to discuss the potential causes and cures, which I will get to below.
However, not everyone accepted that narrative. Publishing in PNAS in 2018, Daniele Fanelli reports:
This article provides an overview of recent evidence suggesting that this narrative is mistaken, and argues that a narrative of epochal changes and empowerment of scientists would be more accurate, inspiring, and compelling.
Everyone seems to agree that there a problem of unreliable research in every field of science, but disagree on the scope and impact. Fanelli argues that the problem is minor, laying out the evidence for this position:
Recent evidence from metaresearch studies suggests that issues with research integrity and reproducibility, while certainly important phenomena that need to be addressed, are: (i) not distorting the majority of the literature, in science as a whole as well as within any given discipline; (ii) heterogeneously distributed across subfields in any given area, which suggests that generalizations are unjustified; and (iii) not growing, as the crisis narrative would presuppose.
On the other end of the spectrum is Dorothy Bishop, who in 2019, again in Nature, stated:
I strongly identify with the movement to make the practice of science more robust. It’s not that my contemporaries are unconcerned about doing science well; it’s just that many of them don’t seem to recognize that there are serious problems with current practices. By contrast, I think that, in two decades, we will look back on the past 60 years — particularly in biomedical science — and marvel at how much time and money has been wasted on flawed research.
This is an issue we have explored at length here at SBM, and so I am very familiar with the underlying literature, having summarized it many times. My take is that both Fanelli and Bishop are correct – reproducibility and robustness in science is a serious problem, it’s just not a fatal problem.
When I lecture about this issue myself I have to be careful to strike this nuance. When you lay out all the documented pitfalls in science it can give the false impression that science is “broken” or in “crisis”. Not necessarily. Rather, I think science is messy; it takes a winding and inefficient path, but does usually get to its destination eventually. Results that are consistent with reality do have an inherent advantage within science over results that are not in accord with reality, and this advantage does tend to sort them out over time.
The narrative that I prefer is that science is hard and if we want to maximize scientific advance (the results we get for our investment), then we need to optimize the technology of science itself. Methodological problems give us temporary false positives (and sometimes false negatives – but there is a huge bias toward the false positive) and increase the time and resources it takes to get to a reliable answer.
But increased rigor and care cost time and resources also. We could avoid false results in science if we slowed it down to a crawl, but that would defeat the purpose. What is that purpose? That’s actually a good place to start. It is not, I would argue, to minimize error in science, it is to optimize advance, defining advance as producing highly reliable (which includes being correct) conclusions about how the universe works.
In order to discover new things we need to take risks, ask questions, and be willing to be wrong a lot. We need to throw a lot of stuff up against the wall and see what sticks, and then confirm and reconfirm our results. Somewhere there is a sweet spot, where scientific methods are efficient enough to ask a lot of questions, but careful enough to produce reliable results and not go down too many rabbit holes.
Like Fanelli I do not think we need to embrace a “crisis” narrative, and I don’t think science is broken. But like Bishop I think we are currently not at the sweet spot, and need to push methodology toward the rigorous end of the spectrum to get to a better balance than where we are generally today.
There are two other layers here worth mentioning. The first is that not all scientific research has to strike the same balance of efficiency and rigor. We can (and do) take a staged approach. We have preliminary research which is down and dirty, optimized for efficiency with the minimal rigor necessary to produce worthwhile results. Research at this end of the spectrum is designed only to inform later research, not to form a basis for firm conclusions. At the confirmatory end of the spectrum research prioritizes extreme methodological rigor, trying as best as we can to answer the question – is this phenomenon really real?
What is important is that we understand where we are with any given question along this spectrum, and not confuse preliminary research with confirmatory research. I don’t think this is generally a problem within mainstream science (scientists get this), but I do think it is a problem with science reporting to the public. I also think it is a systemic problem within those disciplines that I would consider on the fringe, or pseudoscientific.
Take acupuncture, for example. Normally any particular scientific question will progress from more preliminary to more rigorous research methodology. As the research gets more and more rigorous, we begin to see the real effect size or reality of the phenomenon. When we get to the most rigorous studies, we can start to form reliable conclusions. Acupuncture research, to a degree, progressed toward more rigorous studies. The problem (for acupuncturists) is that these studies are generally negative.
Acupuncturists, rather than coming to the correct scientific conclusion that acupuncture does not work, instead started doing less rigorous research. They retreated to more preliminary designs, which they disguised as “pragmatic” studies, and just kept cranking out false positive preliminary research. That is where acupuncture research now lives.
This also relates to the second layer worth mentioning – some sciences are applied. Medicine, for example, is not just engaged in the abstract process of understanding biology. We need answers that determine how we actually treat patients in the real world. In applied sciences we need to have, I argue, a nuanced and sophisticated understanding of the nature of scientific research and its relationship to practice. When are results reliable enough to affect our practice? Where should we draw the line? This, of course, is the focus of most of the articles here on SBM.
Finally, while there is disagreement over the extent and implications of the replication problem, there is broad agreement on the causes and some of the fixes. Everyone agrees that we need to minimize fraud and outright error. Bishop colorfully designates the main problems as the:
…four horsemen of the reproducibility apocalypse: publication bias, low statistical power, P-value hacking and HARKing (hypothesizing after results are known).
Every researcher needs to have a deep understanding of these phenomena and how to avoid them. The original Nature survey asked scientists about what potential fixes they think would be most worthwhile. Here they are (from most to least popular): Better understanding of statistics, better mentoring, more robust design, better teaching, more within-lab replication, incentives for better practice, incentives for formal reproduction, more external lab validation, more time for mentoring, journals enforcing standards, and more time checking notebooks.
It’s hard to argue that these are not all good things. The only objection raised is that these cost more time and resources, but I do think the evidence supports a cultural shift within science toward more rigor and methods for minimizing error. Every researcher, for example, needs to know exactly what P-hacking is and how to avoid it.
It also seems that journals need to value replications more, and negative results more. Bishop points out, for example, that a negative study is often referred to as a “failed” study. This is completely wrong – if the study was rigorous and achieved a reliable result, it did not fail. It answered the question, the answer just happens to be no. A failed study is one that is a false positive or negative because of poor methodology.
At the very least there needs to be better education of scientists and applied science practitioners about all these issues. I routinely lecture to my colleagues about SBM and always take the opportunity to ask basic questions, like “Who knows what p-hacking is? Or how about funnel plots?” I often get mostly blank stares with just a few hands going up. I do think we’re making progress, and the fact that mainstream science journals are publishing opinion pieces about this issue is encouraging. But we have a ways to go – particularly as we get farther and farther out to the fringes. That is where so-called “alternative medicine” lives and why it is such a focus of SBM.

Medical Reversals – Checking the Evidence
How often are standard medical practices reversed by later high quality evidence?
Steven Novella on February 16, 2022
Link
Science-based medicine is partly an endeavor to marshal all the available evidence about medical evidence itself. Modern medicine is ultimately about applying scientific evidence to practice, therefore we constantly have to make specific decisions about that evidence. We need to understand what the science says, and not just in the abstract but with regard to practice recommendations and specific decisions with individual patients.
Part of this science-based decision making is to think very carefully about the threshold of evidence required before it is reasonable to adopt a specific medical practice. Of course, this is very context-dependent. If someone has an incurable fatal illness a lower threshold may be appropriate, compared to a self-limiting symptomatic illness. The guiding principle is to consider the entire package of risk/expense/inconvenience to potential benefit. Evidence also has to be put into the context of overall scientific plausibility – the less plausible a potential intervention, the more evidence is needed to conclude that it likely works or even may work.
This is a complex and individual decision, but we want it to be informed by science and evidence as much as possible. And again, that is what SBM is – examining the complex relationship between scientific evidence (from basic science to clinical studies) and clinical practice. Toward that end we need to not only examine the scientific evidence but to also carefully calibrate where the threshold of adoption should be. One way to inform that calibration is to look back at current medical practice and see how often new practices are reversed by later higher-quality evidence (so-called medical reversals).
A number of studies have done just that. A 2013 study by Prasad et.al. looked at 1,344 studies in a high-impact medical journal from 2001 to 2010 and found:
A total of 947 studies (70.5%) had positive findings, whereas 397 (29.5%) reached a negative conclusion. A total of 756 articles addressing a medical practice constituted replacement, 165 were back to the drawing board, 146 were medical reversals, 138 were reaffirmations, and 139 were inconclusive. Of the 363 articles testing standard of care, 146 (40.2%) reversed that practice, whereas 138 (38.0%) reaffirmed it.
That last line in the most important – of the studies that were examining a practice that was already adopted as part of the standard of care, 40.2% were reversed (the evidence showed the practice did not work), while only 38% showed the practice did work (the rest were inconclusive). This does not mean that 38% of what physicians do is not evidence-based or doesn’t work. This was not a review of practice, but a review of published studies. The fact that medical scientists felt it was necessary to study a practice indicates that it was in question in the first place.
But still – 40% seems high. It suggests that perhaps we have been setting the threshold for adoption of new practices too low. What would the ideal number be? That’s a good question. It should not be zero, because that would likely mean we are setting the threshold too high and leaving potentially useful interventions on the table. It should not be 100%, because that would imply a serious problem with preliminary evidence and that we are setting the threshold way too low. Something in the single digits for reversals is likely closer to ideal, considering that these are practices that have been incorporated into the standard of care.
As I said, however, there is another variable here – what is the threshold for conducting research to question an established practice? Maybe researchers are really good at singling out those practices which need to be questioned. Perhaps high-impact journals have a selection-bias toward publishing such reversals.
Other research on reversals also give different outcomes, likely based on the above variables, in addition to different specialties perhaps having a different calculus regarding risk vs benefit. A 2019 study by Prasad and others found:
Through an analysis of more than 3000 randomized controlled trials (RCTs) published in three leading medical journals (the Journal of the American Medical Association, the Lancet, and the New England Journal of Medicine), we have identified 396 medical reversals.
A 2021 article looking at primary care practice found:
We evaluated 408 POEMs (Patient Oriented Evidence that Matters) on RCTs. Of those, 35 (9%; 95% confidence interval [6-12]) were identified as reversed, 359 (88%) were identified as not reversed, and 14 (3%) were indeterminate. On average, this represents about 2 evidence reversals per annum for POEMs about RCTs.
That seems like a more reasonable figure for reversals – 9% – but still, as the authors points out, that leads to a steady number of medical practices that get reversed. In a 2020 article Haslam, Livingston, and Prasad (the same Prasad from the 2013 article above, and several articles on SBM) examined practices that had been reversed. They found:
We recently identified almost 400 medical practices that were used in clinical care before they were tested in well-done randomized controlled trials and subsequently were found to be ineffective or harmful.
Although these practices were implemented because of sound biologic plausibility or encouraging observational data, well done randomized controlled trials have failed to show evidence of effectiveness. These examples raise caution in introducing new clinical interventions into widespread clinical practice without sufficient high-quality evidence demonstrating efficacy.
That supports the conclusion (one which we make frequently here) that a major cause of medical reversals is adopting practice based on preliminary evidence alone. Practitioners often underestimate how profound research bias can be and how unreliable preliminary quality clinical trials can be. In fact, more often than not, encouraging preliminary evidence is not supported by later high-quality clinical trials.
How long are such practices in use before they are found to be wanting? In a 2021 article looking specifically at oncology journals, they found:
The median number of years that the practice had been in use prior to the reversal study was 9 years (range 1–50 years).
That is a long time, and likely represents many patients. The upper end is most interesting – 50 years of clinical use before being reversed.
The totality of evidence suggests that we can and should do better. There is some low-hanging fruit here – stop basing practices on preliminary evidence that has not been confirmed with a reasonably high-quality clinical trial. We can carve out an exception here for compassionate use of experimental treatments in certain clinical contexts, but the vast majority of uses would not qualify for this exception. There is clear evidence, painstakingly reviewed in the pages of SBM and elsewhere, that most preliminary studies in retrospect will be wrong, and that there is a bias toward creating, publishing, and citing positive studies over negative studies. And more recently we know on the back end that many of these practices will be later reversed if they are adopted too soon.
The short answer therefore is that we need to set the threshold higher.

The problem with preclinical research? Or: A former pharma exec discovers the nature of science
David Gorski on April 23, 2012
Link
If there’s one thing about quacks, it’s that they are profoundly hostile to science. Actually, they have a seriously mixed up view of science in that they hate it because it doesn’t support what they believe. Yet at the same time they very much crave the imprimatur that science provides. When science tells them they are wrong, they therefore often try to attack the scientific method itself or claim that they are the true scientists. We see this behavior not just in quackery but any time scientific findings collide with entrenched belief systems, for example, medicine, evolution, anthropogenic global warming, and many others. So it was not surprising that a rant I saw a few weeks ago by a well-known supporter of pseudoscience who blogs under the pseudonym of Vox Day caught my interest. Basically, he saw a news report about an article in Nature condemning the quality of current preclinical research. From it, he draws exactly the wrong conclusions about what this article means for medical science:
Fascinating. That’s an 88.6 percent unreliability rate for landmark, gold-standard science. Imagine how bad it is in the stuff that is only peer-reviewed and isn’t even theoretically replicable, like evolutionary biology. Keep that figure in mind the next time some secularist is claiming that we should structure society around scientific technocracy; they are arguing for the foundation of society upon something that has a reliability rate of 11 percent.
Now, I’ve noted previously that atheists often attempt to compare ideal science with real theology and noted that in a fair comparison, ideal theology trumps ideal science. But as we gather more evidence about the true reliability of science, it is becoming increasingly obvious that real theology also trumps real science. The selling point of science is supposed to be its replicability… so what is the value of science that cannot be repeated?
No, a problem with science as it is carried out by scientists in the real world doesn’t mean that religion is true or that a crank like Vox is somehow the “real” intellectual defender of science. Later, Vox doubles down on his misunderstanding by trying to argue that the problem in this article means that science is not, in fact, “self-correcting.” This is, of course, nonsense in that the very article Vox is touting is an example of science trying to correct itself. Be that at it may, none of this is surprising, given that Vox has demonstrated considerable crank magnetism, being antivaccine, anti-evolution, an anthropogenic global warming denialist, and just in general anti-science, but he’s not alone. Quackery supporters of all stripes are jumping on the bandwagon to imply that this study somehow “proves” that the scientific basis of medicine is invalid. A writer at Mike Adams’ wretched hive of scum and quackery, NaturalNews.com, crows:
Begley says he cannot publish the names of the studies whose findings are false. But since it is now apparent that the vast majority of them are invalid, it only follows that the vast majority of modern approaches to cancer treatment are also invalid.
But does this study show this? I must admit that it was a topic of conversation at the recent AACR meeting, given that the article was published shortly before the meeting. It’s also been a topic of e-mail conversations and debates at my very own institution. But do the findings reported in this article mean that the scientific basis of cancer treatment is so off-base that quackery of the sort championed by Mike Adams is a viable alternative or that science-based medicine is irrevocably broken?
Not so fast there, pardner…
A systemic problem with preclinical research? Maybe. Maybe not.
One of the most difficult aspects of science to convey to the general public about science-based medicine (and science in general) is just how messy it is. Scientists know that early reports in the peer-reviewed literature are by their very nature tentative and have a high probability of ultimately being found to be incorrect. Unfortunately, that is not science as it is imbibed by the public. Fed by too-trite tales of simple linear progressions from observation to theory to observation to better theory taught in school, as well as media portrayals of scientists as finding answers fast, most people seem to think that science is staid, predictable, and able to generate results virtually on demand. This sort of impression is fed even by shows that I kind of like for their ability to excite people about science, for instance CSI: Crime Scene Investigation and all of its offspring and many imitators. These shows portray beautiful people wearing beautiful pristine lab coats back lit in beautiful labs using perfectly styled multicolored Eppendorf tubes doing various assays and getting answers in minutes that normally take hours, days, or sometimes weeks. Often these assays are all done over a backing soundtrack consisting of classic rock or newer (but still relatively safe) “alternative” rock. And that’s just for applied science, in which no truly new ground is broken and no new discoveries are made.
Real scientists know that cutting edge (or even not-so-cutting edge) scientific and medical research isn’t like that at all. It’s tentative. It might well be wrong. It might even on occasion be spectacularly wrong. But even results that are later found to be wrong are potentially valuable.
Sometimes moviemakers and TV producers get it close to right in showing how difficult science is. For example the HBO movie Something The Lord Made showed just how difficult it could be to take a scientifically plausible hypothesis and turn it into a treatment. In most movies, TV shows, and popular writings, the retrospectoscope makes it seem as though what we know now flowed obviously from the observations of scientific giants. Meanwhile, the news media pounces on each new press release describing new studies as though each was a breakthrough, even though the vast majority of new studies, even seemingly interesting ones, fade into obscurity, to be replaced by the next new “breakthrough.”
In the real world of science, however, things are, as I said, messy. What amazes me is how two scientists, one of whom I respect, can fall prey to amazement when they point out just how messy science is. I’m referring to a commentary that appeared in Nature three weeks ago by C. Glenn Begley, a consultant for Amgen, and Lee M. Ellis, a cancer surgeon at the University of Texas M.D. Anderson Cancer Center. The article was entitled, unimaginatively enough, Drug development: Raise standards for preclinical cancer research. This article is simultaneously an indictment of preclinical research for cancer and a proposal for working to correct the problems identified. It is also simultaneously disturbing, reassuring, and, unfortunately, more than a little misguided.
Before I get into the article, let me just expound a bit (or pontificate or bloviate, depending on what you think of my opinionated writing) about preclinical research. Preclinical research is, by definition, preclinical. It’s the groundwork, the preliminary research, that needs to be done to determine the plausibility and feasibility of a new treatment before testing it out in humans. As such, preclinical research encompasses basic research and translational research and can include biochemical, cell culture, and animal experiments. Depending on the nature of the problem and proposed treatment, it could also include chemistry, engineering, and surgical research.
Now here’s the pontification and bloviation. These days, everybody touts “translational” research, meaning research that is designed to have its results translated into human treatments. It’s darned near impossible these days to get a pure basic science project funded by the NIH; there has to be a translational angle. Often this leads basic scientists to find rather—shall we say?—creative ways of selling their research as potentially having a rapid clinical application, even though they know and reviewers know that such an application could be a decade away. Indeed, if we are to believe John Ioannidis, the median time from idea to completion of large scale clinical trials needed to approve a new treatment based on that idea is on the order of one to two decades. Moreover, as I’ve said many times before, translational research will grind to a halt if there isn’t a robust pipeline of basic science research to provide hypotheses and new biological understandings to test in more “practical” trials. A robust pipeline is necessary because the vast majority of discoveries that look promising in terms of resulting in a therapy will not pan out. That is the nature of science, after all. Many leads are identified; few end up being a treatment.
Not surprisingly, this nature of science seems to be what concerns Begley and Ellis. They begin by pointing out:
Sadly, clinical trials in oncology have the highest failure rate compared with other therapeutic areas. Given the high unmet need in oncology, it is understandable that barriers to clinical development may be lower than for other disease areas, and a larger number of drugs with suboptimal preclinical validation will enter oncology trials. However, this low success rate is not sustainable or acceptable, and investigators must reassess their approach to translating discovery research into greater clinical success and impact.
Of course, some of the reason that clinical trials in oncology have a high failure rate is no doubt due to the high difficulty of the disease (actually many diseases) being tackled. As I’ve pointed out time and time again, cancer is very, very complicated and very, very hard. Given that challenge, as frustrating as it is, it is probably not surprising that only around 5% of agents found to have anticancer activity in preclinical experiments go on to demonstrate sufficient efficacy in phase III clinical trials to earn licensing for sale and use. That is compared to approximately 20% for cardiovascular disease. Of course, cardiovascular drugs are targeted at cells that are nowhere near as messed up as cancer cells, and another study cited by Begley and Ellis suggests that between 20-25% of important preclinical results can’t be reproduced in pharmaceutical company laboratories with sufficient rigor to go forward. Even so, being scientists, we want to improve the process. To improve the process, however, we need to know where the process fails.
To try to do this, Begley and Ellis looked at 53 “landmark” publications in cancer. Begley used to be head of global cancer research at Amgen and knows what it takes to get a drug from idea to market. What it takes first is replication. Basically, Begley’s team would scour the scientific literature for interesting and promising results and then try to replicate them in such a way that their results could serve as a basis for developing drugs based on them. The idea was to identify new molecular targets for cancer and then figure out ways to make drugs to target them. This is what he reported:
Over the past decade, before pursuing a particular line of research, scientists (including C.G.B.) in the haematology and oncology department at the biotechnology firm Amgen in Thousand Oaks, California, tried to confirm published findings related to that work. Fifty-three papers were deemed ‘landmark’ studies (see ‘Reproducibility of research findings’). It was acknowledged from the outset that some of the data might not hold up, because papers were deliberately selected that described something completely new, such as fresh approaches to targeting cancers or alternative clinical uses for existing therapeutics. Nevertheless, scientific findings were confirmed in only 6 (11%) cases. Even knowing the limitations of preclinical research, this was a shocking result.
Here’s the part that I found to be profoundly misguided. Begley and Ellis basically admit that these are “landmark papers”; i.e., that they were highly novel. Presumably these papers would have been considered at the time of their publication to be “cutting edge” research, very likely published in high impact journals such as Nature, Cell, Science, Cancer Research, and the like. Unfortunately, although I looked, I didn’t see a list of the 53 “landmark papers—not even in an online supplement. Nor was the method of how these papers were analyzed described in much detail—not even in an online supplement. The irony inherent in a paper that rails against the irreproducibility of preclinical cancer research but does not itself provide the data upon which its authors based its conclusions in sufficient detail for the reader to determine for himself whether the conclusions flow from the data is left for SBM readers to assess for themselves. Similarly misguided, as was pointed out in the online comments, were the authors’ stated assumption that “the claims in a preclinical study can be taken at face value — that although there might be some errors in detail, the main message of the paper can be relied on and the data will, for the most part, stand the test of time” and their amazement that “this is not always the case.” If the authors’ assumptions were true, attempts to replicate scientific results would be less important than they are.
Be that as it may, what the authors are studying, however they studied it and whatever the 53 studies they examined were, is essentially frontier science. Given that, it strikes me as rather strange that they are so amazed that much of the science at the very frontiers turns out not to be correct when tested further. We in science understand the difference between settled textbook science and the sort of frontier science that makes it into journals like Science. Indeed, we often lament that the very highest tier journals, such as Nature, Science, and Cell, tend to be too enamored of publishing what seems to be “sexy science,” exciting or counterintuitive results that really grab the attention of scientists — in other words “cutting edge” or frontier science. Such journals seem to pride themselves on publishing primarily such work (which is one reason why they are so widely read and cited), while more solid, less “sexy” results seem to end up in second-tier journals.
This leads to a paradox: the science that is published in the highest profile, most prestigious journals is almost by definition the most tentative science. Given that, it is surprising how much of what is published in such journals actually does stand the test of time, but it should not be surprising that much of it does not. However, the very prestige of such journals gives such research seemingly more authority than research published in less prestigious journals. It is often said that one Nature, Science, or Cell paper is worth five or even ten papers in more pedestrian, middle-of-the-road journals when it comes to improving a scientist’s CV (and chance of a good job or promotion). Perhaps that is because publications in such journals are viewed as an indication that the work a scientist is doing is on the cutting edge. That perception, built up over time, is likely the major reason that it is very, very difficult to get a paper accepted and published in Science, Nature, or Cell. The vast majority of submissions are rejected, many without even being sent out for peer review, because an editorial decision is made that they are not “interesting” enough (something that happened to me once). In other words, the editors of such journals are actively looking for science that challenges the existing paradigm. However, scientists understand that papers published in the most cutting edge journals are tentative. They’re interested in the papers because such work is the most likely to advance the frontiers of science.
In fact, they’re interested in such papers for the very same reason that Begley and his group at Amgen were interested in them. Begley was the head of a major research division of a major pharmaceutical company. What does that mean? It means that it was his job to find new molecular targets for cancer and to develop drugs to target them. And it was his job to do all this and beat his competitors to the market with effective new drugs based on these discoveries. No wonder his group scoured high impact journals for cutting edge studies that appeared to have identified promising molecular targets! Then he had a veritable army of scientists, about 100 of them in the Amgen replication team according to this news report, who were ready to pounce on any published study that suggested a molecular target the company deemed promising.
Here’s another aspect of the study that needs to be addressed. As I read the study, a thought kept popping into my fragile eggshell mind. Remember Reynold Spector? He’s the guy whom both Mark Crislip and I jumped on for a particularly bad criticism of science-based medicine and its alleged lack of progress that Spector called Seven Deadly Medical Hypotheses. As both Mark and I pointed out, nearly all of these hypotheses were really not particularly deadly, and, indeed, most of them weren’t even hypotheses. What Dr. Spector shares in common with Dr. Begley is a background in pharma, and the similarities in the way they think are obvious to all. For instance, I castigated Spector for throwing around the term “pseudoscience” to describe studies that in his estimation do not reach the level of evidence necessary for FDA approval of a drug. That is a very specific set of requirements for a very specific problem: developing a drug from first scientific principles and then demonstrating that it is efficacious for the intended indication as well as safe. I got the impression from his articles that Dr. Spector views any study that doesn’t reach FDA-level standards for drug approval to be pseudoscience — or, at the very least, crap. I get the same impression from Begley. For example, here’s a passage from his article:
Of course, the validation attempts may have failed because of technical differences or difficulties, despite efforts to ensure that this was not the case. Additional models were also used in the validation, because to drive a drug-development programme it is essential that findings are sufficiently robust and applicable beyond the one narrow experimental model that may have been enough for publication.
Elsewhere in the article, Begley defines “non-reproduced” as a term he assigned “on the basis of findings not being sufficiently robust to drive a drug-development programme.” This attitude is, of course, understandable in someone running an oncology drug development program for a major pharmaceutical company. He is looking for results that he can turn into FDA-approved drugs that he can bring to market before his competitors do. So what he does is more than just try to reproduce the results as described in the publication. His team of 100 scientists tries to reproduce the results and extend them to multiple model systems relevant to drug design. That is, in essence, applied science. Think of it this way: How many basic science discoveries in physics and chemistry ever get turned into a product? How many of these findings are sufficiently robust and reproducible in multiple model systems to justify a team of engineers to spend millions of dollars developing them into products? Do physicists, materials scientists, chemists, and engineers obsess over how few findings in basic science in their fields can successfully be used to make a product?
I know, I know, apples and oranges. In medicine, those of us doing research do it in order to develop an understanding of a disease process sufficient to develop an efficacious new treatment. It’s a very explicit in what we do. However, sometimes we forget just how important it is to have a large, robust pipeline of preclinical results upon which to base translational research programs. Is the reason for the apparently declining percentage of basic science studies that are successfully translated into drugs more a function of the increasing ability of scientists, through large scale genomic and small molecule screens, to identify more and more potential molecular targets and potential drugs to use against them than of scientists doing something wrong? I also have to wonder if what Begley and Ellis are observing is the decline effect accelerated by 100 scientists prowling the scientific literature looking for experimental results they can turn into drugs. As I pointed out before, the decline effect doesn’t mean science doesn’t work, and, as I will point out here, Begley’s very methods would almost be expected to accelerate the decline effect.
The rest of the story
Don’t get me wrong. Although I find the premise of Begley and Ellis’ article to be misguided, there is important and disturbing information there. Unfortunately, the really important and disturbing information is not in Begley and Ellis’ paper. The omission of these critical pieces of information strikes me as a curious decision on the part of the authors and Nature editors.
For example, in the paper, we learn this:
In studies for which findings could be reproduced, authors had paid close attention to controls, reagents, investigator bias and describing the complete data set. For results that could not be reproduced, however, data were not routinely analysed by investigators blinded to the experimental versus control groups. Investigators frequently presented the results of one experiment, such as a single Western-blot analysis. They sometimes said they presented specific experiments that supported their underlying hypothesis, but that were not reflective of the entire data set. There are no guidelines that require all data sets to be reported in a paper; often, original data are removed during the peer review and publication process.
This is one reason that when I review papers I always ask if assays were performed in a blinded fashion, particularly when the results involve selecting parts of histological slides for any sort of quantification.
In an interview, however, we learn a lot more critical information:
When the Amgen replication team of about 100 scientists could not confirm reported results, they contacted the authors. Those who cooperated discussed what might account for the inability of Amgen to confirm the results. Some let Amgen borrow antibodies and other materials used in the original study or even repeat experiments under the original authors’ direction.
Some authors required the Amgen scientists sign a confidentiality agreement barring them from disclosing data at odds with the original findings. “The world will never know” which 47 studies — many of them highly cited — are apparently wrong, Begley said.
I find it very interesting that Begley didn’t mention this rather important tidbit of information in the Nature paper and why he and Ellis didn’t see fit to name names of studies for which non-disclosure agreements weren’t signed. One wonders if he (and the Nature editors) were concerned about litigation. In any case, the non-disclosure agreements obviously must predate the Nature paper. This tells me that Begley was in essence complicit in not revealing that his team couldn’t reproduce results, apparently not thinking such agreements to be too high a price at the time for access to reagents and help in the cause of advancing his company’s efforts. He’s willing to admit this in news interviews, apparently, but not in the Nature paper being used as a broadside against current preclinical drug development efforts.
Here’s another highly irritating passage from Begley and Ellis’ paper:
Some non-reproducible preclinical papers had spawned an entire field, with hundreds of secondary publications that expanded on elements of the original observation, but did not actually seek to confirm or falsify its fundamental basis. More troubling, some of the research has triggered a series of clinical studies — suggesting that many patients had subjected themselves to a trial of a regimen or agent that probably wouldn’t work.
Why do I say this is an “irritating” passage? Simple. It would have been very helpful if Begley and Ellis had actually named a couple of these “entire fields,” don’t you think? I suppose they probably couldn’t do that without indirectly revealing which papers whose results Begley’s team couldn’t reproduce. The lack of this information makes this jeremiad against how preclinical research is done today far less useful for actually fixing the problem than it might have been. Assessing the irony of a paper railing against current preclinical research methods that does not itself reveal its methods in sufficient detail to be evaluated or even its results except in fairly vague ways is again left as an exercise for SBM readers.
There are also many explanations for the variability in published research, as has been pointed out by other commentators. For instance, Nobel Laureate Phil Sharp homes in on one problem:
The most common response by the challenged scientists was: “you didn’t do it right.” Indeed, cancer biology is fiendishly complex, noted Phil Sharp, a cancer biologist and Nobel laureate at the Massachusetts Institute of Technology.
Even in the most rigorous studies, the results might be reproducible only in very specific conditions, Sharp explained: “A cancer cell might respond one way in one set of conditions and another way in different conditions. I think a lot of the variability can come from that.”
It’s true, too. I remember back in the late 1990s, several labs were having difficulty reproducing Judah Folkman’s landmark work on angiogenesis inhibitors, including the lab where I was working at the time. Dr. Folkman provided reagents, protocols, and advice to any who asked, and ultimately we were able to find out what the problem was, part of which was that the peptide we were using was easily denatured. We also learned that he had done the same thing for several labs, even to the point of dispatching one of his postdocs to help other investigators. Now imagine if Folkman had been like one of the scientists who had demanded non-disclosure agreements when Begley’s group had trouble reproducing his studies.
Still, the problems with Begley and Ellis’ article notwithstanding, they do provide useful information and identify what appears to be a serious problem. The problem is not so much that so few basic science discoveries end up as drugs, courtesy of Amgen or one of its big pharma competitors. Rather, it’s the sloppiness that is too common in the scientific literature, coupled with publication bias, investigator biases, and the proliferation of screening experiments done to identify genomic targets and small molecules with biological effects that has turned into the proverbial fire hose of data, often many terrabytes per screen. I also wonder if part of the problem is that all the “easy” molecular targets for therapy have already been identified, leaving the difficult and problematic ones. The result is alluded to but not adequately discussed in the news story I cited above:
As recently as the late 1990s, most potential cancer-drug targets were backed by 100 to 200 publications. Now each may have fewer than half a dozen.
The genomics, proteomics, and metabolomics revolutions that have occurred over the last 10-15 years are largely to blame for this. I would also argue (and perhaps Begley would even agree) that the competitiveness between pharmaceutical companies to be the “firstest with the mostest” for each new target hyped in the medical literature almost certainly contributes to this problem. After having been burned a few times, Begley could, for instance, have decided that his team wouldn’t seize on each of these new papers, that he’d wait until some more papers were published. He didn’t do that. For him, business as usual continued. An admission that he was part of the problem, either in the Nature paper or one of the interviews he gave to the press, would have been nice.
How to improve preclinical research
It’s true that I’ve been critical of Begley and Ellis’ article, but that’s mainly because of frustration. There are many things that need to be improved in terms of how science is applied. Readers might recall that I’ve written about problems with the peer review system, publication bias, the decline effect, and numerous other problems that interfere with the advancement of science and contribute to doubts about its reliability. Such problems are inevitable because science is done by humans, with all their biases, cognitive quirks, and conflicts of interests, but that doesn’t mean every effort shouldn’t be made to minimize them. Science remains the single best system for determining how nature works, and, no matter how much quacks and cranks might try to cast doubt on it because it doesn’t support their pseudoscience, no one has as yet developed a better system.
The question, therefore, is how to minimize the effects these problems have on how the scientific method is practiced, particularly given that the scientific method itself is designed to try to minimize the effects of human shortcomings on how evidence is gathered and analyzed. No matter how much cranks like Vox Day and Mike Adams’ minions try to portray Begley and Ellis’ article as an indictment of science itself, as slam dunk evidence that that science is not self-correcting and the scientific basis of cancer therapy is so much in doubt that quackery is a viable alternative or that religion is a more reliable way of seeking knowledge about the world than science, it is in fact nothing of the sort.
It does, however, tell us that we as scientists need to improve, and, indeed, we at SBM have discussed the shortcomings of medical science and ways to improve upon it on many occasions. In fact, I daresay that much of what we say jibes with the suggestions proposed by Begley and Ellis, including:
    • More opportunities to present negative data.
    • An agreement that negative data can be as informative as positive data.
    • Requiring preclinical investigators to present all findings.
    • Links added to articles to other studies that show different or alternate results.
    • Transparent opportunities for trainees, technicians and colleagues to discuss and report troubling or unethical behaviours without fearing adverse consequences.
    • “Greater dialogue should be encouraged between physicians, scientists, patient advocates and patients. Scientists benefit from learning about clinical reality. Physicians need better knowledge of the challenges and limitations of preclinical studies. Both groups benefit from improved understanding of patients’ concerns.”
    • More credit for teaching and mentoring.
    • Less emphasis on publication in top-tier journals.
    • “Funding organizations must recognize and embrace the need for new cancer research tools and assist in their development, and in providing greater community access to those tools. Examples include support for establishing large cancer cell-line collections with easy investigator access (a simple, universal material-transfer agreement); capabilities for genetic characterization of newly derived tumour cell lines and xenografts; identification of patient selection biomarkers; and generation of more robust, predictive tumour models.”
Many of these are good ideas, although I’m not sure how practical it would be to require that investigators present “all” findings in journal articles and how such a requirement would be enforced. Defining “all” would be a challenge, and online supplements are already too much of a dumping ground these days. For example, does “all” mean investigators have to present the dozens of attempts it might have required to optimize assay conditions or include every experiment that was screwed up because someone used the wrong conditions or added the wrong reagent or left their tubes sit on the bench too long? Also, one notes how Begley assiduously avoids criticizing pharma for being so eager to leap on the latest cutting edge research before it has percolated through the literature, which, I conclude based on his very own complaint, is surely part of the problem.
So is the very nature of science. Scientists know that what is published the first time is considered tentative. It may or may not be correct. We also know that publication bias can mean that the first publication of a result might well be an anomaly that was published because it was interesting. That is science at the frontier. If other scientists can replicate the results or, even better, replicate the results and use them as a foundation to build upon and make new discoveries, only then does such a result become less frontier science. And if the results are replicated enough times and by enough people and used as a basis for further discoveries, to the point that they are considered settled results, that’s when they become applied science, such as a drug based on the principle originally discovered. It’s a process that is very messy and with lots of dead ends and blind alleys that go nowhere. While performing a valuable service that identifies problems with a lack of reproducibility in all too many preclinical cancer research studies, Begley and Ellis also unfortunately contribute to the mistaken impression that translational research is a linear process that goes from discovery to drug. It’s not, nor can it churn out major new treatments on demand.

   Being Negative Is Not So Bad
Steven Novella
Link
A new study published in PLOS Biology looks at the potential magnitude and effect of publication bias in animal trials. Essentially, the authors conclude that there is a significant file-drawer effect – failure to publish negative studies – with animal studies and this impacts the translation of animal research to human clinical trials.
SBM is greatly concerned with the technology of medical science. On one level, the methods of individual studies need to be closely analyzed for rigor and bias. But we also go to great pains to dispel the myth that individual studies can tell us much about the practice of medicine.
Reliable conclusions come from interpreting the literature as a whole, and not just individual studies. Further, the whole of the literature is greater than the sum of individual studies – there are patterns and effects in the literature itself that need to be considered.
One big effect is the file-drawer effect, or publication bias – the tendency to publish positive studies more than negative studies. A study showing that a treatment works or has potential is often seen as doing more for the reputation of a journal and the careers of the scientists than negative studies. So studies with no measurable effect tend to languish unpublished.
Individual studies looking at an ineffective treatment (if we assume perfect research methodology) should vary around no net effect. If those studies that are positive at random are more likely to be published than those studies that are neutral or negative, than any systematic review of the published literature is likely to find a falsely positive effect.
Of course, we do not live in a perfect world and many studies have imperfect methods and even hidden biases. So in reality there is likely to be a positive bias to the studies. This positive bias magnifies the positive publication bias.
There are attempts in the works to mitigate the problem of publication bias in the clinical literature. For example, clinicaltrials.gov is a registry of all trials involving human subjects – before the trials are completed and the results known. This way reviewers can have access to all the data – not just the data researchers and journal editors deem worthy.
This new study seeks to explore if publication bias is similarly a problem with animal studies. The issues are similar to human trials. There is an ethical question, as sacrificing animals in research is justified by the data we get in return. If that data is hidden and does not become part of the published record, than the animals were sacrificed for nothing.
And also, publication bias can lead to false conclusions. This in turn can, for example, lead to clinical trials of a drug that seems promising in animal studies. This could potentially expose human subjects to a harmful or just worthless drug that would not have made it to human trials if all the negative animal data were published.
The study itself looked at a database of animal models of stroke. They examined 525 publications involving 16 different stroke interventions. There are a few different types of statistical analysis that can be done to infer probable publication bias. Basically, without publication bias there should be a certain distribution of findings in terms of effect sizes. If only positive or larger effect sizes are being published, then the distribution will be skewed.
This type of analysis provides an estimation only. They found that:
Egger regression and trim-and-fill analysis suggested that publication bias was highly prevalent (present in the literature for 16 and ten interventions, respectively) in animal studies modelling stroke. Trim-and-fill analysis suggested that publication bias might account for around one-third of the efficacy reported in systematic reviews, with reported efficacy falling from 31.3% to 23.8% after adjustment for publication bias. We estimate that a further 214 experiments (in addition to the 1,359 identified through rigorous systematic review; non publication rate 14%) have been conducted but not reported. It is probable that publication bias has an important impact in other animal disease models, and more broadly in the life sciences.
So there was some disagreement between the methods used, but both showed that there is likely to be a significant publication bias. If their analysis is correct, about one third of systematic reviews of animal studies in stroke that conclude an intervention works may be due to publication bias rather than a real effect. The authors also speculate that this effect is likely not unique to stroke, and may be generalizable to animal studies in general.
Of course, this is just an individual study, and further analysis using different data sets are needed to confirm these results.
Conclusion
The results of this study are not surprising and are in line with what is known from examining clinical trials. They suggest that similar methods to minimize publication bias are necessary for animal studies in addition to human trials.
Hopefully, this kind of self-critical analysis will lead to improvement in the technology of medical research. It should further lead to more caution in interpreting not only single studies but systematic reviews.
Also, in my opinion, it highlights the need to consider basic science and plausibility in evaluating animal and clinical trials.

Open vs Blinded Peer-Review
Steven Novella on September 30, 2015
Link

The overall goal of science-based medicine is to maintain and improve the standard of science in the practice of medicine at every level. At the heart of the scientific basis of medical knowledge and practice is a process known as peer-review. We have occasionally written about peer-review on SBM, and once again the process is under the microscope over a specific question – should peer review be open or blinded?
What is peer review
The term peer-review refers to a pre-publication process in which a journal editor will send a submitted manuscript to 2-3 experts in a field relevant to the paper to carefully examine every aspect of the paper. They then provide a detailed analysis of the paper: is the research question relevant and appropriate, did the study design properly address the question, were the methods rigorous, was the statistical analysis appropriate, was the data presented fairly, are the conclusions supported by the data, did the authors account for other publications addressing the same issue, and did they address every possible question or objection?
The reviewers’ reports will make specific recommendations for changes that would be necessary to improve the paper, and also make their bottom-line recommendation: accept as is, accept with revisions, or deny. The journal editor(s) then rely upon those reviews, plus their own assessment, to make a final decision.
The quality of a scientific journal is directly tied to the quality of peer review, which is a key part of the overall editorial quality.
Open vs blinded peer review
Traditionally most peer review is single-blinded. This means that the reviewers know who the authors are, but the authors do not know who the reviewers are. The reviewers remain anonymous (except to the editors). Authors may suggest reviewers to the editors, but the editors ultimately pick them.
The idea behind blinded reviews is that reviewers will feel free to express their true opinions, without worrying about the political fallout, or making enemies with a colleague if they recommend their paper be rejected. This also avoids tit-for-tat reviews, where a reviewer gives a negative review because one of the authors previously gave one of their papers a bad review, or colleagues exchanging positive reviews.
Some journals are experimenting with an open peer review process, in which the reviewers are publicly known, and their reviews are entirely transparent. In this model the transparency is the mechanism of quality control. The idea is that reviewers will do a thorough and fair job of peer review because all their colleagues will be able to see the job they did. If they were sloppy, or unfairly positive or negative, their peers will see it.
Open peer review should not be confused with open-access peer review. Val Jones wrote about this previously – open access peer review means that anyone can review a paper, which is made available in a public space for anyone to review and comment. This isn’t even really “peer review” – it’s just open access review, because the reviewers don’t have to be peers. This is the crowd-sourcing/Wikipedia approach to peer-review.
As Val pointed out, Peter Frishauf, the founder of Medscape, published a webcast editorial in 2008 predicting that:
Peer review as we know it will disappear. Rather than the secretive prepublication review process followed by most publishers today, including Medscape, most peer review will occur transparently, and after publication.
This seems like a premature prediction. I suspect open-access post-publication review will supplement, and not replace, peer-review. Or, this kind of system may exist parallel to the peer-reviewed published literature.
Head-to-head comparison
There doesn’t seem to be a clear choice between blinded and open peer-review, as both have their strengths and weaknesses. A recent study in BMJ Open directly compared blinded vs open peer review: “Retrospective analysis of the quality of reports by author-suggested and non-author-suggested reviewers in journals operating on open or single-blind peer review models.”
The study compared 200 reviews for articles each from open and blinded journals that were otherwise similar:
BMC Infectious Diseases and BMC Microbiology are similar in size, rejection rates, impact factors and editorial processes, but the former uses open peer review while the latter uses single-blind peer review.
They also did a second comparison. BMC Inflammation was open and then transitioned to blinded peer review, so they compared 200 reviews from the open period with 200 reviews from the blinded period.
In all cases the reviews were independently evaluated and given a standard assessment of quality.
The study also tracked the results of author-suggested reviewers vs other reviewers. What they found was that the quality of reviews submitted by author-suggested reviewers was the same as other reviewers under both models. However, author-suggested reviewers were much more likely to recommend acceptance. They also found that editors put less weight on author-suggested reviewers than other reviewers.
Regarding open vs blinded reviews, the results were mixed. When comparing BMC Infectious Diseases (open) to BMC Microbiology (blinded) they found that the quality of the open reviews on average scored 5% better than the blinded reviews. This result was barely statistically significant at p=0.042.
When comparing BMC Inflammation before and after transitioning from a blinded to an open peer review model, they found no difference in quality.
Conclusion – both methods seem valid
The bottom line of this study is that there does not seem to be much of a difference between the quality of reviews from an open vs blinded peer review model. From the press-release of the study:
Maria Kowalczuk, lead author, BioMed Central’s Biology Editor for the Research Integrity Group and co-Editor-in-Chief of Research Integrity and Peer Review, said: “As advocates of openness, we are excited to find that upon analysis reviewer reports under open peer review are of comparable, or even higher, quality than those of the more established model of single-blind peer review. However, we appreciate our results do not undermine the single-blind model of peer review.
Given that the blinded peer review model is more established, even showing no difference would tend to validate the newer open peer-review model. Of course, this one study does not end the debate, as it was limited to only three journals within one publishing group.
This study also does not address concerns about the downstream effects of open vs blinded review on the culture of science. Do scientists want their reviews to be public, and do they want to know who their reviewers are? How will this affect the community of scientists? These questions are much more difficult to assess.
As the two models seem roughly equivalent (if these results hold up), it does not seem like an objective measure like review quality is going to settle the debate.

   What’s The Harm?
Steven Novella
Link
Any promoter of science-based medicine often faces the question – what’s the harm? What is the harm if people try treatment modalities that are not based upon good science, that are anecdotal, or provide only a placebo benefit? There are generally two premises to this question. The first is that most “alternative” placebo interventions are directly harmless. The second is that direct harm is the only type worth considering. Both of these premises are wrong.
The pages of SBM are filled with accounts of direct harm from unscientific treatments: argyria from colloidal silver, death from chelation therapy, infection or other complications from acupuncture, burns from ear candling, stroke from chiropractic neck manipulation – the list goes on. You can read anecdotal accounts of such harm on the website, whatstheharm.net. Of course, as we often point out, harm and risk is only one end of the equation – one must also consider benefit. It is the risk/benefit ratio of an intervention that is important. But generally we are talking about interventions that lack any evidence for benefit, and therefore any risk of harm is arguably unacceptable.
But perhaps the far greater harm comes from indirect causes. I was reminded of this with the publication of a study looking at flu vaccine uptake in 9 countries. They found that among older individuals who did not get the flu vaccine there was an increase in negative attitudes toward the vaccine, but also there was an increase in the use of traditional unscientific interventions. Cause and effect here is likely to be complicated. People who rely upon folk remedies may feel that they do not need the flu vaccine. Also, those who do not trust in the vaccine may then seek out alternatives. It is likely also true that the subculture of “alternative” medicine simultaneously fosters both a belief in unscientific treatments and a mistrust of mainstream science-based interventions.
The study authors conclude:
The hypothetical framework can be used to guide healthcare providers in developing strategies to foster normative beliefs of older people in vaccination, provide effective action cues and promote vaccine access.
“Normative beliefs” is a technical way of referring to the fact that promoters of “alternative” modalities tend to foster bizarre and unscientific beliefs in the public. Dubious treatments are often marketed with false notions about biology, physiology, and anatomy. It turns out, the iris of the eyes do not contain a functional map of the body (nor does the bottom of the foot). There is no human energy field or biofield or chi. Acupuncture points have not basis in reality. Magnets do not attract the iron in our blood. And toxins do not build up in our tissues, causes most diseases.
The consequences of fostering incorrect notions about human physiology and the nature of health and disease are difficult to measure or quantify, but they should not be ignored as a significant source of indirect harm from unscientific treatments.
The marketing of unscientific treatments often involves warning potential customers away from mainstream medicine, or at least downplaying the effectiveness of science-based treatments or overstating their risks. One does not have to look beyond any pro-CAM website to see articles scaring the public off science-based treatments side by side with advertisements for unscientific alternatives. The internet is unfortunately full of commercialized websites working directly against efforts to create “normative beliefs” in the public.
Belief in ineffective treatments and mistrust of science-based treatments may lead to delay in effective treatment and worse outcomes. In addition they create financial harm, which is increasingly important as health care costs rise. Financial harm can be extreme in cases of the desperation caused by serious illnesses. Tens of thousands of dollars are spent, for example, to send one patient to a fraudulent stem cell clinic. This financial harm is not limited to the patient or their family, as often such treatments are funded by charity from extended family, friends, and colleagues.
There is further psychological harm from creating false hope. I have personally seen the crushing effects such false hope can create when reality finally sets in. This can also significantly delay the process of psychologically dealing with a serious illness, which further affects important decision making about care. Wasting time on worthless treatments can also rob the terminally ill of precious time spent with loved-ones.
Further, the allure of unrealistic treatments diverts scarce resources (hospital space, research time and money) away from more fruitful modalities.
Conclusion
The ripple effect of harm that flows from unscientific medical beliefs is multifarious and significant, but often neglected by those who are not familiar with the phenomenon. “What’s the harm” is therefore a question we will have to answer frequently and for the foreseeable future.

   The “decline effect”: Is it a real decline or just science correcting itself?
Published by David Gorski
Link
‘Tis the season, it would seem, for questioning the scientific method.
You might recall that back in October, I was a bit miffed by an article in The Atlantic entitled Lies, Damned Lies, and Medical Science and expressed my annoyance in one of my typical logorrheic posts. Then, a mere couple of weeks later, Steve Simon wrote a rather scathing criticism of the very concept of science-based medicine, which I ended up answering, again in my usual inimitable logorrheic fashion. Unfortunately, these things often come in threes. Well, maybe not always threes. It’s not as though this “rule” is anything like the count for the Holy Hand Grenade of Antioch, where “Four shalt thou not count, nor either count thou two, excepting that thou then proceed to three. Five is right out.” Except that five isn’t always right out when it comes to these sorts of criticisms of science and/or science-based medicine.
But enough of my pathetic attempt to channel Mark Crislip. The third count in articles expressing skepticism of the scientific method and science-based medicine comes, for purposes of my discussion, in the form of an article in The New Yorker by Jonah Lehrer entitled The Truth Wears Off: Is There Something Wrong With the Scientific Method? Unfortunately, the full article is restricted only to subscribers. Fortunately, a reader sent me a PDF of the article; otherwise, I wouldn’t have bothered to discuss it. Also, Lehrer himself has elaborated a bit on questions asked of him since the article’s publication and published fairly sizable excerpts from his article here and here. In any case, I’ll try to quote as much of the article as I think I can get away with without violating fair use, and those of you who don’t have a subscription to The New Yorker might just have to trust my characterization of the rest. It’s not an ideal situation, but it’s what I have to work with.
The decline effect
I’m going to go about this in a slightly different manner than one might normally expect. First, I’m going to quote the a few sentences near the end of the article right now at the beginning, because you’ll rapidly see why those of us here at SBM might find them provocative, perhaps even a gauntlet thrown down. Before I do that, I should define the topic of the article, namely something that has been dubbed “the decline effect.” Basically, this is a term for a phenomenon in which initial results from experiments or studies of a scientific question are highly impressive, but, over time, become less so as the same investigators and other investigators try to replicate the results, usually as a means of building on them. In fact, Googling “the decline effect” brought up an entry from The Skeptic’s Dictionary, in which the decline effect is described thusly:
The decline effect is the notion that psychics lose their powers under continued investigation. This idea is based on the observation that subjects who do significantly better than chance in early trials tend to do worse in later trials.
In his article, Lehrer actually does cite paranormal research by Joseph Banks Rhine in the 1930s, whose testing of a self-proclaimed psychic demonstrated lots of “hits” early on, far more than were likely to be due to random chance. Initial results appeared to support the existence of extrasensory perception (ESP). However, as further testing progressed, the number of hits fell over time, hence the term “decline effect” being coined to describe it. Lehrer spends the bulk of his article describing examples of the decline effect, discussing potential explanations for this observation, and trying to argue that the effect can be generalized to nearly all of science. Longtime readers of SBM would probably not find that much particularly irksome or objectionable in his article; that is, until we get to the final paragraph:
Such anomalies demonstrate the slipperiness of empiricism. Although many scientific ideas generate conflicting results and suffer from falling effect sizes, they continue to get cited in the textbooks and drive standard medical practice. Why? Because these ideas seem true. Because they make sense. Because we can’t bear to let them go. And this is why the decline effect is so troubling. Not because it reveals the human fallibility of science, in which data are tweaked and beliefs shape perceptions. (Such shortcomings aren’t surprising, at least for scientists.) And not because it reveals that many of our most exciting theories are fleeting fads and will soon be rejected. (That idea has been around since Thomas Kuhn.) The decline effect is troubling because it reminds us how difficult it is to prove anything. We like to pretend that our experiments define the truth for us. But that’s often not the case. Just because an idea is true doesn’t mean it can be proved. And just because an idea can be proved doesn’t mean it’s true. When the experiments are done, we still have to choose what to believe.
As you might imagine, this passage rather irritated me with what appears on the surface to border on a postmodernist rejection of the scientific method as “just another way of knowing” in which we as scientists have to “choose what to believe.” Moreover, it certainly seems that many of the examples provided by Lehrer are compelling and curious examples of effect sizes declining in a variety of scientific areas. What’s not quite as compelling is the way Lehrer, whether intentionally or inadvertently, gives the impression (to me, at least) of painting the decline effect as some sort of mysterious and unexplained phenomenon that isn’t adequately explained by the various explanations he describes in his article and therefore casts serious doubt on the whole enterprise of science in general and SBM in particular, given that many of his examples come from medicine. In all fairness, Lehrer did later try to justify the way he concluded his article. to boil it all down, basically Lehrer equivocated that all he meant by the above passage was that science is “a lot messier” than experiments, clinical trials, and peer review and that “no single test can define the truth.” No kidding. (The snark in me might also say that science itself can’t actually define “The Truth.”) But if that’s all that Lehrer really meant, then why didn’t he just say so in the first place instead of sounding all postmodernist, as though science can’t ever make any conclusions that are any more valid than “other ways of knowing”?
So which examples does Lehrer choose to bolster his case that the decline effect is a serious and underrecognized problem in science? He uses quite a few, several from medical sciences (in particular psychiatry), starting the article out with the example of second generation antipsychotics, such as Zyprexa, which appeared to be so much more effective than older antipsychotics in earlier studies but whose efficacy has recently been called into question, as more recent studies have showed lower levels of efficacy, levels that are no better than the older drugs. Of course Lehrer seems never to have heard of the “dilution effect,” whereby new drugs, once approved, are tried in larger and broader ranges of conditions and patients, in particular, in patients with milder cases of the diseases for which the drugs were designed. Over time, this frequently results in the appearance of declining efficacy, when in reality all that is happening is that physicians and scientists are pushing the envelope testing the drugs in patients who are less carefully selected than patients in the early trials. No real mystery here.
Another example came from evolutionary biology, specifically observations on fluctuating symmetry. This passage is taken from a blog post quoting Lehrer’s article:
In 1991, the Danish zoologist Anders Møller, at Uppsala University, in Sweden, made a remarkable discovery about sex, barn swallows, and symmetry. It had long been known that the asymmetrical appearance of a creature was directly linked to the amount of mutation in its genome, so that more mutations led to more “fluctuating asymmetry.” (An easy way to measure asymmetry in humans is to compare the length of the fingers on each hand.) What Møller discovered is that female barn swallows were far more likely to mate with male birds that had long, symmetrical feathers. This suggested that the picky females were using symmetry as a proxy for the quality of male genes. Møller’s paper, which was published in Nature, set off a frenzy of research. Here was an easily measured, widely applicable indicator of genetic quality, and females could be shown to gravitate toward it. Aesthetics was really about genetics.
In the three years following, there were ten independent tests of the role of fluctuating asymmetry in sexual selection, and nine of them found a relationship between symmetry and male reproductive success. It didn’t matter if scientists were looking at the hairs on fruit flies or replicating the swallow studies—females seemed to prefer males with mirrored halves. Before long, the theory was applied to humans. Researchers found, for instance, that women preferred the smell of symmetrical men, but only during the fertile phase of the menstrual cycle. Other studies claimed that females had more orgasms when their partners were symmetrical, while a paper by anthropologists at Rutgers analyzed forty Jamaican dance routines and dis- covered that symmetrical men were consistently rated as better dancers.
Then the theory started to fall apart. In 1994, there were fourteen published tests of symmetry and sexual selection, and only eight found a correlation. In 1995, there were eight papers on the subject, and only four got a positive result. By 1998, when there were twelve additional investigations of fluctuating asymmetry, only a third of them confirmed the theory. Worse still, even the studies that yielded some positive result showed a steadily declining effect size. Between 1992 and 1997, the average effect size shrank by eighty per cent.
And it’s not just fluctuating asymmetry. In 2001, Michael Jennions, a biologist at the Australian National University, set out to analyze “temporal trends” across a wide range of subjects in ecology and evolutionary biology. He looked at hundreds of papers and forty-four meta-analyses (that is, statistical syntheses of related studies), and discovered a consistent decline effect over time, as many of the theories seemed to fade into irrelevance. In fact, even when numerous variables were controlled for — Jennions knew, for instance, that the same author might publish several critical papers, which could distort his analysis—there was still a significant decrease in the validity of the hypothesis, often within a year of publication. Jennions admits that his findings are troubling, but expresses a reluctance to talk about them publicly. “This is a very sensitive issue for scientists,” he says. “You know, we’re supposed to be dealing with hard facts, the stuff that’s supposed to stand the test of time. But when you see these trends you become a little more skeptical of things.”
Jennions’ article was entitled Relationships fade with time: a meta-analysis of temporal trends in publication in ecology and evolution. Reading the article, I was actually struck by how relatively small, at least compared to the impression that Lehrer gave in his article, the decline effect in evolutionary biology was found to be in Jennions’ study. Basically, Jennions examined 44 peer-reviewed meta-analyses and analyzed the relationship between effect size and year of publication; the relationship between effect size and sample size; and the relationship between standardized effect size and sample size. To boil it all down, Jennions et al concluded, “On average, there was a small but significant decline in effect size with year of publication. For the original empirical studies there was also a significant decrease in effect size as sample size increased. However, the effect of year of publication remained even after we controlled for sampling effort.” They concluded that publication bias was the “most parsimonious” explanation for this declining effect.
Personally, I’m not sure why Jennions was so reluctant to talk about such things publicly. You’d think from his responses in Lehrer’s interview that scientists would be coming for him with pitchforks, hot tar, and feathers if he dared to point out that effect sizes reported by investigators in his scientific discipline exhibit small declines over the years due to publication bias and the bandwagon effect. Perhaps it’s because he’s not in medicine; after all, we’ve been speaking of such things publicly for a long time. Indeed, we generally expect that most initially promising results, even in randomized trials, will not ultimately pan out. In any case, those of us in medicine who might not have been willing to talk about such phenomena became more than willing after John Ioannidis published his provocatively titled article Why Most Published Research Findings Are False around the time of his study Contradicted and Initially Stronger Effects in Highly Cited Clinical Research. Physicians and scientists are generally aware of the shortcomings of the biomedical literature. Most, but sadly not all of us, know that early findings that haven’t been replicated yet should be viewed with extreme skepticism and that we can become more confident in results the more they are replicated and built upon, particularly if multiple lines of evidence (basic science, clinical trials, epidemiology) all converge on the same answer. The public, on the other hand, tends not to understand this.
John Ioannidis, John Lehrer, and the decline effect, and me
Of course, the work of John Ioannidis, as discussed here before, most recently by myself, provides an excellent framework to understand why effect sizes appear to decline over time. Although Ioannidis has been criticized for exaggerating the extent of the problem and even using circular reasoning, for the most part I find his analysis compelling. In medicine, in particular, early reports tend to be smaller trials and experiments that, because of their size, tend to be more prone to false positive results. Such false positive results (or, perhaps, exaggerated results that appear more positive than they really are) generate enthusiasm, and more investigators pile on. There’s often a tendency to want to publish confirmatory papers early on (the “bandwagon effect”), which might further skew the literature too far towards the positive. Ultimately, larger, more rigorous studies are done, and these studies result in a “regression to the mean” of sorts, in which the newer studies fail to replicate the large effects seen in earlier results. This is nothing more than what we’ve been writing right here on SBM ever since its inception, namely that the normal course of clinical research is to start out with observations from smaller studies, which are inherently less reliable because they are small and thus more prone to false positives or exaggerated effect sizes
In his article, Lehrer blames in essence three things for the decline effect: publication bias, selective reporting, and the culture of science, which contributes to the proliferation of the first two problems. Publication bias has been discussed here on SBM on multiple occasions and in various contexts. Basically, it’s the phenomenon in which there is a marked bias towards the publication of “positive” data; in other words, negative studies tend not to be reported as often or tend to end up being published in lower tier, lower “impact” journals. To Lehrer, however, publication bias is not adequate to explain the decline effect because, according to him:
While publication bias almost certainly plays a role in the decline effect, it remains an incomplete explanation. For one thing, it fails to account for the prevalence of positive results among studies that never even get submitted to journals. It also fails to explain the experience of people like Schooler, who have been unable to replicate their initial data despite their best efforts.
This is what is known about being (probably) right for the wrong reasons. I would certainly agree that publication bias is probably an incomplete explanation for the decline effect, although I would be very curious about the prevalence of positive results among studies that never get submitted to journals; it’s pretty darned rare, in my experience, for positive results not to be submitted for publication unless there are serious flaws in the studies with positive results or some other mitigating circumstance takes hold, such as the death of the principal investigator, a conflict over the results between collaborating laboratories, or a loss of funding that prevents the completion of necessary controls or additional experiments. If Lehrer has evidence that show my impression that failure to publish positive results is rare, he does not present it.
I would also argue that Lehrer is probably only partially right (and makes a huge assumption to boot) when he argues that publication bias fails to explain why individual investigators can’t replicate their own results. Such investigators, it needs to be remembered, initially published highly positive results. When they have trouble showing effect sizes as large and seemingly robust as their initial results, doubt creeps in. Were they wrong the first time? Will reviewers give them a hard time because their current results do not show the same effect sizes as their original results? They hold back. True, this is not the same thing as publication bias, but publication bias contributes to it. A journal’s peer reviewers are probably going to give an investigator a much harder time for a result showing a smaller effect size if there is published data from before that shows a much larger effect size; better journals will be less likely to publish such a result, and investigators know it. Consequently, publication bias and selective reporting (the investigator holding back the newer, less compelling results, knowing the lower likelihood of getting it published in a top tier journal). Other investigators, not invested in the original investigator’s initial highly positive results, are less likely to hold back, and, indeed, there may even be an incentive to try to disprove a rival’s results.
Lehrer makes a good point when he points out that there is such a thing as selective reporting, wherein investigators tend to be less likely to report findings that do not fit into their current world view and might even go so far as to try to shoehorn findings into the paradigm they currently favor. He even goes so far as to give a good example of cultural effects on selective reporting, specifically the well-known tendency of studies of acupuncture from China to be far more likely to report positive results than studies of acupuncture done in “Western” nations. He points out that this discrepancy “suggests that scientists find ways to confirm their preferred hypothesis, disregarding what they don’t want to see.” Or, as Simon and Garfunkel once sang in The Boxer, “a man hears what he wants to hear and disregards the rest.” It is not surprising that scientists would share this quality with their fellow human beings, but it is devilishly difficult to identify and quantify such biases. That, of course, doesn’t stop proponents of pseudoscience from crying “bias!” whenever their results are rejected by mainstream science.
There is one other potential explanation that Lehrer seems not to consider at all: Popularity. About a year and a half ago, I discussed a fascinating study that examined the effect of popularity on the reliability of the medical literature about a topic. The study, by Pfeiffer and Hoffman, was published in PLoS ONE and entitled Large-Scale Assessment of the Effect of Popularity on the Reliability of Research, and its introduction lays out the problem:
In this context, a high popularity of research topics has been argued to have a detrimental effect on the reliability of published research findings [2]. Two distinctive mechanisms have been suggested: First, in highly competitive fields there might be stronger incentives to “manufacture” positive results by, for example, modifying data or statistical tests until formal statistical significance is obtained [2]. This leads to inflated error rates for individual findings: actual error probabilities are larger than those given in the publications. We refer to this mechanism as “inflated error effect”. The second effect results from multiple independent testing of the same hypotheses by competing research groups. The more often a hypothesis is tested, the more likely a positive result is obtained and published even if the hypothesis is false. Multiple independent testing increases the fraction of false hypotheses among those hypotheses that are supported by at least one positive result. Thereby it distorts the overall picture of evidence. We refer to this mechanism as “multiple testing effect”. Putting it simple, this effect means that in hot research fields one can expect to find some positive finding for almost any claim, while this is not the case in research fields with little competition [1], [2].
I discussed the implications of this paper in my usual nauseating level of detail here. Suffice to say, the more scientists working on a problem there are, the more false positives there are likely to be, but, as the field matures, there is a regression to the mean. Also, don’t forget that initial exciting results are often published in the “highest” impact journals, publication in which can really make a scientist’s career take off. However, because these results are the most provocative and might even challenge the scientific consensus strongly, they also have a tendency to turn out later to be wrong. Leaving out this aspect is a major weakness in Lehrer’s analysis, particularly given that each of the examples he provided could easily have a major component of the “popularity effect” going on.
The bottom line: Is the scientific method unreliable?
As I read Lehrer’s article, I was troubled. No, I wasn’t troubled because the implications of his article were somehow shaking my view of the reliability of science and the scientific method. I certainly wasn’t troubled by his discussing known problems with how science is practiced by fallible human beings, how it almost always isn’t done completely according to the idealized version of the scientific method taught to us in high school. After all, I’ve discussed the problems of publication bias and deficiencies in the peer review system seemingly ad nauseam. Rather, I was troubled by the final paragraph, quoted above, in which Lehrer seems to be implying, if not outright arguing, that science is nothing more than competing narratives between which scientists must choose, each of them not particularly well supported by data. Jerry Coyne nails it when he comments:
But let’s not throw out the baby with the bathwater. In many fields, especially physics, chemistry, and molecular biology, workers regularly repeat the results of others, since progress in their own work demands it. The material basis of heredity, for example, is DNA, a double helix whose sequence of nucleotide bases codes (in a triplet code) for proteins. We’re beginning to learn the intricate ways that genes are regulated in organisms. The material basis of heredity and development is not something we “choose” to believe: it’s something that’s been forced on us by repeated findings of many scientists. This is true for physics and chemistry as well, despite Lehrer’s suggestion that “the law of gravity hasn’t always been perfect at predicting real-world phenomena.”
Lehrer, like Gould in his book The Mismeasure of Man, has done a service by pointing out that scientists are humans after all, and that their drive for reputation—and other nonscientific issues—can affect what they produce or perceive as “truth.” But it’s a mistake to imply that all scientific truth is simply a choice among explanations that aren’t very well supported. We must remember that scientific “truth” means “the best provisional explanation, but one so compelling that you’d have to be a fool not to accept it.” Truth, then, while always provisional, is not necessarily evanescent. To the degree that Lehrer implies otherwise, his article is deeply damaging to science.
Indeed. There is no such thing as scientific “truth,” actually. In fact, one thing I noticed right away in Lehrer’s articles is that the examples he chose were, by and large, taken from either psychology, parapsychology, or ecology, rather than physics and chemistry. True, he did point out how most gene association studies with diseases thus far have not been confirmed and how different groups find different results, but finding such associations is something that is currently popular but not a mature field. According to the “popularity effect,” it is not surprising that there is currently a lot of “noise” out there in terms of scientific results. Over the next decade, it is very likely that many of these questions and disagreements will be sorted out scientifically.
Finally, Lehrer’s view also seems not entirely consistent in some ways. I’ll show you what I mean. On his blog, as I mentioned before, Lehrer answers reader questions and expands upon his ideas a bit. A reader asks Lehrer, “Does this mean I don’t have to believe in climate change?” Lehrer’s response is, basically, that “these are theories that have been verified in thousands of different ways by thousands of different scientists working in many different fields,” which is, of course, true, but almost irrelevant given Lehrer’s previous arguments. After all, even though I accept the scientific consensus regarding anthropogenic global warming, if publication bias and selective reporting can so distort science for so long in other fields, I have to ask how would Lehrer say he accepts the science of global warming. One way is that he quite correctly points out that the “truths” of science (I really hate using that word with respect to science) depend upon the strength of the “web” supporting them, namely the number of interconnections. We say that here ourselves time and time again as arguments against pseudoscience such as, for example, homeopathy. However, if, as Lehrer seems to be arguing, scientists already put their results into the context of what is known before, isn’t he just basically arguing for doing what we are already doing, even though he has just criticized science for being biased due to selective reporting due to scientists’ existing preconceptions?
Although Lehrer makes some good points, where he stumbles, from my perspective, is when he appears to conflate “truth” with science or, more properly, accept the idea that there are scientific “truths,” even going so far as to use the word in the title of his article. That is a profound misrepresentation of the nature of science, in which all “truths” are provisional and all “truths” are subject to revision based on evidence and experimentation. The decline effect–or, as Lehrer describes it the title of his article, the “truth wearing off”–is nothing more than science doing what science does: Correcting itself.
Maybe the “mysterious decline effect” isn’t nearly as mysterious as Lehrer implies.

   Rambling Musings on Using the Medical Literature
Mark Crislip
Link
For those who are new to the blog, I am nobody from nowhere. I am a clinician, taking care of patients with infectious diseases at several hospitals in the Portland area. I am not part of an academic center (although we are affiliated with OHSU and have a medicine residency program). I have not done any research since I was a fellow, 20 years ago. I was an excellent example of the Peter Principal; there was no bench experiment that I could not screw up.
My principal weapon in patient care is the medical literature, accessed throughout the day thanks to Google and PubMed. The medical literature is enormous. There are more than 21,000,000 articles referenced on Pubmed, over a million if the search term ‘infection’ is used, with 45,000 last year.
I probably read as much of the ID literature as any specialist. Preparing for my Puscast podcast, I skim several hundred titles every two weeks, usually select around 80 references of interest and read most of them with varying degrees of depth. Yet I am still sipping at a fire hose of information
The old definition of a specialist is someone who knows more and more about less and less until they everything about nothing. I often feel I know less and less about more and more until someday I will know nothing about everything. Yet I am considered knowledgeable by the American Board of Internal Medicine (ABIM), who wasted huge amounts of my time, a serious chunk of my cash, and who have declared, after years of testing, that I am recertified in my specialty. I am still Board Certified, but the nearly pointless exercise has left me certified bored. But I can rant for hours on Bored Certification and how out of touch with the practice of medicine the ABIM is.
My concept of an expert is a combination of experience and understanding of the literature. I used to say mastery of the literature, but no one can master a beast that large; I am just riding on the Great A’Tuin of medical writings. Experience comes with time, and I have read that it takes 10 years to become competent in a field. Whether true or not, it matches my experience. I remember as a resident reading notes on patients I had cared for as an intern, and being appalled with what an ignorant dufus I was. In my first year of practice I had a patient who died of miliary tuberculosis, and the diagnosis was, unfortunately, made at autopsy. It was an atypical manifestation of a rare (in the US) disease. About a decade later the case was presented as an unknown to a visiting professor; I had completely forgotten the case, but I piped up from the audience to pontificate on how this had to be miliary Tb. Afterwards I was shown the chart and nice documentation as to how clueless I had been a decade earlier. When it comes to being a diagnostician, there is not substitute for experience.
When it comes to treatment? That is where I tell the residents that the three most dangerous words in medicine are ‘In. My. Experience.’ You cannot trust experience when deciding on therapy, especially for relatively unusual diseases. Sometimes I will ask a doc why they use a given antibiotic, usually in a situation where it is being used in a way that is, shall we say, old fashioned. Often the response is “I like it”‘ as if the choice of a drug is like choosing a beer.
I rely on the literature — such as it is, and limited by my lack of an Ethernet jack in my brain — in deciding the best course of therapy for a patient. The literature is always unsatisfactory. That has always been known. Even with the best studies, there is always the issue of wondering if the literature applies to your patient and their particular co-morbidities, and, perhaps, genetics. As an example, it is becoming evident that the literature on the presentation and treatment of Cryptococcus, which is based on the experience with C. neoformans, is not applicable to C. gattii, a new strain of the fungus in the NW. So how to use a literature that may not be totally relevant to my local conditions? I wing it. It is an educated and experienced winging, but winging it I do.
Given the breadth and depth of the literature, it is nice to have systematic reviews, meta-analysis, and guidelines. As a practicing physician, I find them helpful as they provide an overarching understanding, a conceptual framework, for understanding a disease or a treatment. They are the Reader’s Digest abridged version of a topic, and the references are invaluable. Usually most of the relevant literature is collected in these reviews and make it easier, especially in the era of the Googles and on-line references, to find the original literature.
All three have their flaws, and if you are well versed in a field, you recognize the issues and try and compensate.
As was noted in the recent Archives, the literature to support the recommendations of the Infectious Disease Society America are not necessarily based on the best of evidence. Really? I’m shocked. Next up, water is wet, fire is hot, and the Archives confirms the obvious.
Results In the 41 analyzed guidelines, 4218 individual recommendations were found and tabulated. Fourteen percent of the recommendations were classified as level I, 31% as level II, and 55% as level III evidence. Among class A recommendations (good evidence for support), 23% were level I (1 randomized controlled trial) and 37% were based on expert opinion only (level III). Updated guidelines expanded the absolute number of individual recommendations substantially. However, few were due to a sizable increase in level I evidence; most additional recommendations had level II and III evidence.
Conclusions: More than half of the current recommendations of the IDSA are based on level III evidence only. Until more data from well-designed controlled clinical trials become available, physicians should remain cautious when using current guidelines as the sole source guiding patient care decisions.
Big duh. Anyone who is a specialist understands the weaknesses in all guidelines, but we also understand their importance. When I was a fellow, one of my attending was, and still is, one of the foremost experts in the US on Candida, and anothers areas of expertise is S. aureus infections and endocarditis.
Both have spent a career thinking deeply on their respective areas of expertise. You learn that while no one is perfect, the breadth and depth of their knowledge and experience gives their recommendations extra weight. Who would you want at the controls of your plane in a unexpected and unusual weather conditions? An experienced pilot, or someone who spent a few days on the X-Plane simulator? The same with all the guidelines. When someone with a lifetimes of work in a field helps write a guideline, you pay attention to their expertise. You know the recommendations are not necessarily right, but odds are their opinions are better than mine, just as my opinion is usually better than a hospitalist, as least as far as infections are concerned. With residents, I try make a point of differentiating when my recommendation is no better than the next doc, and when my recommendation is the Truth, big T, and based on the best understanding of the literature at the moment.
This attitude, trusting authority, held by many in medicine, goes against the University of Google approach where a day of searching and a quick misreading of the abstracts renders everyone an expert. I wonder if other fields are plagued with these quick pseudo-experts. Law is, when the accused attempt to defend themselves.
I certainly would be in favor of more money being spent on infectious disease research, and, one hopes, infectious disease doctors. In a perfect world, every disease would be subjected to careful, extensive clinical trials and I would know, for example, the best therapy for invasive Aspergillus pneumonia in a neutropenic leukemia patient. Until that time, I am, in part, going to rely on the guidelines written by those who have spent a career thinking about the diseases I have to treat. To quote Dr Powers,
“Guidelines may provide a starting point for searching for information, but they are not the finish line…Evaluating evidence is about assessing probability,” Dr. Powers commented in a news release. “Perhaps the main point we should take from the studies on quality of evidence is to be wary of falling into the trap of ‘cookbook medicine,’” Dr. Powers continues. “Although the evidence and recommendations in guidelines may change across time, providers will always have a need to know how to think about clinical problems, not just what to think.”
I was struck by a recent Medscape headline:
Cochrane Review Stirs Controversy Over Statins in Primary Prevention
Having been irritated of late by Cochrane reviews in my area of expertise, I clicked the link. The first three paragraphs are
A new Cochrane review has provoked controversy by concluding that there is not enough evidence to recommend the widespread use of statins in the primary prevention of heart disease.
The authors of the new Cochrane meta-analysis, led by Dr Fiona Taylor (London School of Hygiene and Tropical Medicine, UK), issued a press release questioning the benefit of statins in primary prevention and suggesting that the previous data showing benefit may have been biased by industry-funded studies. This has led to headlines in many UK newspapers saying that the drugs are being overused and that millions of people are needlessly exposing themselves to potential side effects.
This has angered researchers who have conducted other large statin meta-analyses, who say the drugs are beneficial, even in the lowest-risk individuals, and their risk of side effects is negligible. They maintain that the Cochrane reviewers have misrepresented the data, which they say could have serious negative consequences for many patients currently taking these agents.
Newsweek and The Atlantic both refer to the Cochrane review as a “study.” A review is not what I would consider a study, usually synonymous with a clinical trial. The use of the term makes it sound like the Cochrane folks were doing a clinical trial, patients being randomized to one treatment or another. My sloppy non scientific poll of people (all people in the medical field, but that is who I have contact with) suggests the no one considers a review of clinical trials to be a study. A review of a novel is not the same as writing a novel.
Sloppy and potentially misleading language from major news outlets. What a surprise.
I have always liked meta-analysis for the same reason I like guidelines: they provide an overarching conceptual framework for understanding a topic. But only a fool would make clinical decisions based upon a meta-analysis. Yet, meta-analysis seem to be creeping to the top of the list of the clinical information rankings to be believed.
There are issues with meta-analysis.
The studies included in a meta-analysis are often of suboptimal quality. Many spend time bemoaning the lack of quality studies they are about to stuff into their study grinder. Then, despite knowing that the input is poor quality, the go ahead and make a sausage. The theory, as I said last week, is that if you collect many individual cow pies into one big pile, the manure transmogrifies into gold. I still think it as a case of GIGO: Garbage In, Garbage Out.
It has always been my understanding that a meta-analysis was used in lieu of a quality clinical trial. Once you had a few high quality studies, you could ignore the conclusions of a meta-analysis.
Evaluations of the validity of the conclusions of meta-analysis have demonstrated that the results of a meta-analysis usually fail to predict the results of future good clinical trials. The JREF million is safe from the Cochrane, I suppose. Their conclusions are no more reliable than the studies they collect and are no more valid than the rest of the medical literature.
We identified 12 large randomized, controlled trials and 19 meta-analyses addressing the same questions. For a total of 40 primary and secondary outcomes, agreement between the meta-analyses and the large clinical trials was only fair (kappa= 0.35; 95 percent confidence interval, 0.06 to 0.64). The positive predictive value of the meta-analyses was 68 percent, and the negative predictive value 67 percent. However, the difference in point estimates between the randomized trials and the meta-analyses was statistically significant for only 5 of the 40 comparisons (12 percent). Furthermore, in each case of disagreement a statistically significant effect of treatment was found by one method, whereas no statistically significant effect was found by the other.
Once there was a quality definitive trial or three, the meta-analysis becomes, I thought, moot. A quality clinical trial trumps the meta. I guess. I am not so certain that is the attitude anymore given the freak-out in the media about Cochrane and statins.
It seems that the producers of meta-analysis have characteristics like the March of Dimes. Polio was conquered, but rather than folding up their tents and stealing away, they continue to march. That may be a good thing too, as there could be a polio resurgence if some anti-vaccine wackaloons have their way.
If there is a definitive trial, rather than declaring the question settled, the new, perhaps higher-quality, study is folded in with the prior studies and a new meta-analysis is generated. But newer studies are diluted by the older, less robust trials, so the more reliable results are lost in the wash. The best drowned in a sea of mediocrity.
For example, I see no need for a meta-analysis on the efficacy of Echinacea. The last several trials, combined with basic science/prior probability, provides sufficient evidence to conclude Echinacea does not work. Good trials win. Ha.
As a practicing specialist, no matter how much I read, I rely in part on guidelines, meta-analyses and systematic reviews as nice overviews to be used as flawed stopgaps awaiting large high quality clinical trials, that, like Godot, may never come.
I have sick patients who need treatment. I need to know what to do. I have to fight the battles with the weapons I have. I have the medical literature and I am not afraid to use it.

   Changing the Rules of Evidence
Steven Novella
Link
My daughter, Julia, loves to play games and has a bit of a competitive streak. She can make any activity into a game and is adept at making up rules on the spot. When she was younger, like most children, she had a tendency to add to or change the rules on the fly – usually to ensure a favorable outcome for herself. “Oh, Daddy, I forgot to mention that the ball can bounce once and that still counts.”
It was an opportunity for me to gently teach her that in order for rules to work everyone has to know what they are ahead of time and you can’t change them after the fact. Her smile told me that even at five she intuitively knew this already – that changing or making up new rules was not fair. What I was really teaching her was that she wasn’t going to get away with it with me, and by extension that it is socially unacceptable to mess with the rules to suit oneself.
Adults are really no different than children in our basic emotional makeup. We all want to change the rules to suit our own needs. The true difference is that as we mature we become more socially sophisticated; we become more subtle in our manipulations, and we develop the capacity to rationalize our wants and desires. We also learn that we are playing a bigger game – the social game. So we adhere to the rules of fairness, even if it means losing a competition, because we want to succeed at the more important game of socialization. (I’m not making any moral or ethical judgments here, just observing human behavior.)
Rules don’t just exist for games. Important systems have rules also, often ones that have evolved and developed over time to serve a particular purpose. For example, our democracy has rather complex rules. Political parties have their own rules for how they choose their nominee for president. Right now Hillary Clinton is desperately trying to change those rules in light of the results that have already emerged. She wants the popular vote to have the most importance, including those states (Florida and Michigan) who were deprived of their delegates for breaking the rules. Does anyone doubt for a moment that she is favoring this interpretation because it is the one favorable to her? Does anyone think that a year ago Senator Clinton would have argued that the popular vote was more important than the delegate count? This is nothing more than a more sophisticated version of my daughter’s flexible ball-game rules.
Science also has rules – very important rules. The goal of science is to develop models of the natural world that are reliable, that make predictions about future observations and experiments that turn out to be true. When we spend millions of dollars to launch a probe to Saturn, we want to be rewarded with beautiful pictures of the ringed planet. When we administer a medical treatment we want to be confident that we are more likely to help our patient than to harm them. With science, the stakes are high, so the rules are important.
Mostly the rules of science developed in order to limit our tendency to fool ourselves. That’s right – humans are remarkably adept at self-deception. Our perceptions are biased by our beliefs and desires. We are prone to confirmation bias, which means we notice and remember information that confirms what we already believe, and ignore or rationalize away disconfirming evidence. Our memories are based mostly on emotion – not fidelity of detail. Our intuitive logic is often flawed or oversimplified.
Science is simply a set of rules of investigation. The biggest rule of science is that we have to test our ideas against reality. We can’t just make stuff up and then assume we are right – we have to subject our guesses to observations that have the potential of proving them wrong. Scientific observations must be recorded objectively so that we don’t have to rely upon flawed memories. Outcomes need to be specified ahead of time – we cannot decide at the end of an experiment which results prove our hypothesis. Outcomes should be quantified as much as possible, and as objectively and unambiguously as possible.
Over time the rules of science became more rigorous. We decided that those judging the outcome of an experiment should be blinded, so that their biases could not influence their measurements. In medical studies we decided that double-blind studies were better, because when subjects know they are getting a real treatment they may report their outcomes differently or behave differently. Statistical methods developed to analyze the data mathematically, and cutoffs for significance became generally accepted.
The rules of science are still evolving – they are not written in stone. Also it is often challenging to figure out how to apply the rules of science to new types of questions. But there are many general principles of scientific investigation that hold in any context, and in general the more rigorously the rules of science are followed the more reliable the results.
Today there is a political/ideological movement within medicine and health care to change the rules after the fact. The purveyors of many sectarian methods of treatment and unscientific belief systems of health and illness have not succeeded at the fair rules of science. So now they want to change those rules. They want anecdotes to not only count but to trump rigorously controlled observations (that is, when the anecdotes are in their favor). They was to reinterpret the placebo effect after the fact as if it were a real effect. They want to count only those experiments that confirm their beliefs and ignore or reject those studies that reject their beliefs.
Being educated adults they have much more sophisticated language to express their childish desire to alter the rules.
Andrew Weil wants to relabel anecdotes he favors as “uncontrolled clinical observations.” This is a way of getting to choose after the fact which observations count, rather than letting the rules of science decide.
Dr. David Katz from Yale’s “Integrative Medicine” Program wants to allow for “a more fluid concept of evidence.” This way modalities he favors, such as homeopathy, that have failed by the generally accepted rules of science can still win with his more “fluid” rules.
When studies of “alternative” modalities are negative, proponents want to change the rules after they see the results. They claim that the “sham” acupuncture was giving a real effect too, or that the numbers in the study were too small, or that homeopathy cannot be tested with the same methods as cookie cutter drugs, or that a statistically insignificant trend in their favor should count even though the rules say they shouldn’t. Of course, when the outcome is positive, then these same rules are just fine. Heads I win, tales you lose.
When Congress created the National Center for Complementary and Alternative Medicine (NCCAM) they wanted to create a funding center within the NIH so that researchers who failed to get funding for their research through the existing rigorous rules could succeed with more relaxed rules made especially for them.
Many states in the US now have “healthcare freedom” laws. These laws essentially say that when practitioners fail to abide by the rules of ethical scientific medicine, it’s OK. They can have their own set of “alternative” rules in which they can do whatever they want without having to worry about that pesky “standard of care” or basing their diagnoses and treatments on tiresome “scientific evidence.”
As my daughter intuitively understood at age five – the rules only mean anything if everyone knows the rules ahead of time, if everyone has to play by the same rules, and if the rules cannot be changed ad hoc to favor one side.
The rules of science and evidence exist for a reason – they maximize the reliability of the conclusions we reach. In medicine, an applied science, this has immediate implications for the health of the public. We cannot have two systems of rules in medicine, one for”mainstream” medicine, and a second set of “fluid” rules for everything that fails to meet these rigorous mainstream rules. That is equivalent to having no rules at all. In fact it is worse, because it gives the false impression that there are rules, and this engenders a faith in the system that is not justified.
Unless there is one set of rules the entire system fails. We should not acquiesce to those who want to let the ball bounce once and still count.

   Pragmatic Studies – More Bait and Switch
Steven Novella 
Link
The course of research into so-called alternative medicine (CAM) over the last 20 years has largely followed the same pattern. There was little research into many of the popular CAM modalities, but proponents supported them anyway. We don’t need science, they argued, because we have anecdotes, history, and intuition.
When media attention, which drove public attention, was increasingly paid to CAM then serious scientific research increased. A specific manifestation of this was the National Center for Complementary and Alternative Medicine (NCCAM). CAM proponents then argued that their modalities were legitimate because they were being studied (as if that’s enough). Just you wait until all the positive evidence comes rolling in showing how right we were all along.
But then the evidence started coming in negative. A review of the research funded by NCCAM, for example, found that 10 years and 2.5 billion dollars of research had found no proof for any CAM modality. They must be doing something wrong, Senator Harkin (the NCCAM’s major backer) complained. They engaged in a bit of the kettle defense – they argue that the evidence is positive (by cherry picking, usually preliminary evidence), but when it is pointed out to them that evidence is actually negative they argue that the studies were not done fairly. But then when they are allowed to have studies done their way, but still well-controlled, and they are still negative, they argue that “Western science cannot test my CAM modalities.”
But at the same time, they cannot shake the need for scientific evidence to continue to push their modalities into the mainstream. The “your science can’t test my woo” defense only goes so far. So they have put themselves into a pickle – they demanded funding for CAM research, but now have to deal with the fact that the research is largely negative. CAM proponents are mostly not interested in finding what really works, and abandoning what does not work (I have an open challenge to anyone who can point to a CAM modality that was largely abandoned, rejected, or condemned by CAM proponents because of evidence of lack of efficacy). They are interested in using science to support and promote what they already believe works (a cardinal feature of pseudoscience).
And so they have entered the next phase of CAM research – the post-RCT (randomized controlled trial) phase. They have discovered the “pragmatic” study. You have to give it to them for their cleverness. A pragmatic study is meant to be a comparison of treatment options in real-world conditions. It studies treatments that already have proven efficacy from RCTs to see how they work, and how they compare, when applied in the less controlled environment of real-world practice. Pragmatic studies are useful for addressing the weaknesses of RCTs, mainly their somewhat contrived conditions (having strict inclusion criteria, for example).
But pragmatic studies are not efficacy trials themselves. They cannot be used to determine if a treatment works, because they do not control for variables and they are not blinded, so they are susceptible to placebo and non-specific effects. It is an abuse of the pragmatic study design to test a treatment that is not proven or to make efficacy claims based upon them. That has not stopped CAM proponents from doing just that.
In the news recently is just the latest example. Let me tell you the results of the study before I tell you what the study is of – just look at the data (I replaced the name of the treatment in the tables with just the word “treatment”):
This is what you need to know about the trial. There were two groups, treatment and control. Subjects were those with frequent doctor visits for symptoms that have not been diagnosed. While they were randomized, they were completely unblinded – everyone involved with the subjects knew who was getting treated and who wasn’t. At 26 weeks the control group was crossed over to receive treatment – so the difference up to 26 weeks is most important.
On two of the measures, quality of life and general practitioner consultation rates, there was no difference. In the other two measures there was a slight improvement in the treatment group, barely statistically significant. If this were an efficacy trial, this data would be unconvincing. What can fairly be concluded from this trial is that the treatment has no to minimal effect, and the tiny and inconsistent effect seen cannot be separated from placebo or non-specific effects. Further, it shows how anemic even placebo effects are for this treatment in this patient population. As a pragmatic trial, it’s essentially negative. As an efficacy trial, it’s worthless.
The study author’s, however, concluded that their treatment was effective and recommended it be incorporated into general practice.
The treatment in question is five-element acupuncture – acupuncture designed to balance the five elements of fire, water, metal, earth, and wood. This is the equivalent of balancing the four humors – in other words, it is pre-scientific superstitious nonsense. The study authors are all proponents. What is most amazing is how they got their paper, complete with unsupported conclusions, past peer-review.
The study authors are using the “part of this nutritious breakfast” con. Even as a child I understood that the sugary cereal, or whatever the commercial was trying to sell to me, was only “part of this nutritious breakfast” because the rest of the breakfast was nutritious all by itself. A doughnut would be “part of this nutritious breakfast.” So the authors of this study include five-point acupuncture with an hour of kind attention from a practitioner along with lifestyle advice and encouragement. They admit that they cannot separate out the variables here. Then they conclude that acupuncture is “part of this healthy regimen.” There is every reason to believe that it is an irrelevant part – as irrelevant as the doughnut is to the nutrition of a complete breakfast.
Conclusion
The pragmatic study bait and switch is here – it is now a firm part of the CAM strategy for promoting implausible therapies that don’t work. Editors and peer-reviewers need to be more aware of this scam so they don’t fall for it and inadvertently promote it, as they did in this case. Further, the results of this trial are not impressive even at face value – it’s basically negative, so it’s a double swindle.
All of this has not stopped the headlines from declaring that “acupuncture works” – which appears to be the only goal of such research.

   Pragmatic Studies and Cinderella Medicine
Harriet Hall
Link
Explanatory studies are done under controlled conditions to determine whether a treatment has any efficacy compared to a placebo. Pragmatic studies are designed to assess how the intervention performs in everyday real world practice. Pragmatic studies measure practical success but don’t determine actual efficacy: that requires a proper randomized controlled trial (RCT) with an appropriate control. Pragmatic studies have their place, but they can sometimes make an ineffective treatment look good: a phenomenon I have christened Cinderella Medicine.
Research studies don’t always predict how a treatment will perform post-marketing. A study might be done on men aged 30 to 70 with no other illnesses. Once the drug is out on the market, it will probably be taken by women, by people who are under 30 or over 70, and by those who have concurrent diseases like diabetes or atherosclerosis or are taking a lot of other medications; and the dose or frequency may vary from the study protocol.
A good example is the clot-busting drug t-PA. In clinical trials, it clearly improved the outcome of stroke patients, but in actual use in community ERs the death rate was almost twice as high in patients who received it as in patients who did not. There are many possible explanations: perhaps it was offered to patients with more severe strokes, perhaps the protocol wasn’t followed as carefully, perhaps some patients were mis-diagnosed, perhaps they had confounding factors that were not present in the research studies.
Pragmatic studies are useful for the questions they are designed to answer: how a treatment performs outside the limited environment of a research study, which of two treatments should be preferred by doctors, patients, and policymakers, deciding how limited resources can be best utilized. They are an integral part of comparative effectiveness research.
But they have limitations. They are unblinded and the patients are generally self-selected. Larger studies are needed to compare two active treatments than to compare an active treatment to a placebo. Lack of compliance and a high drop-out rate can skew results. Patient self-assessments of subjective outcomes are not as reliable as objectively measurable outcomes.
Pragmatic trials can’t determine which components of a “package” of care are essential, and they can’t assess the contributions of the therapeutic relationship. When used for studying a treatment with a strong placebo effect, they may make an ineffective treatment look better than an effective one. CAM proponents like pragmatic studies because they are often the only studies that seem to support them. They are attempting to bypass good science by showing that, in practice, their methods seem to work.
Cinderella didn’t look very pretty in her rags and ashes. Imagine that her Ugly Stepsister had a complete makeover, with hair styling, expertly applied cosmetics, jewels, and a beautiful designer dress. Maybe tooth whitening or even orthodontia, charm school, modeling classes, and elocution lessons. If you entered her in a beauty contest along with the unadorned, dirty Cinderella in her original rags, the Ugly Stepsister might win hands down. But it wouldn’t be a fair beauty contest unless both were in their original unenhanced state or unless you compared the makeover-enhanced stepsister to the Fairy-Godmother-enhanced Cinderella. (Or if, for a really well-controlled study, you managed to persuade the FG to do her magic on both of them.)
Acupuncture’s makeover
Studies comparing acupuncture to standard treatment have shown that acupuncture works better. Standard treatment is like the original Cinderella in her rags: plain and unenhanced in any way. The doctor may only see the patient for one visit and tell him “You have a common garden variety backache; we don’t know why people get these, but they usually resolve spontaneously in a few weeks; while it is going away on its own, I could offer you a prescription for pain pills or a referral to physical therapy.” He doesn’t spend much time with the patient and may seem bored and unsympathetic.
Acupuncture is like the Ugly Stepsister after her complete makeover. The treatment itself (insertion of needles) is like the Ugly Stepsister before her makeover. It doesn’t have any specific effects (it is no more effective than touching the skin with a toothpick). But the acupuncturist surrounds the treatment with all kinds of enhancements that produce “nonspecific effects” that are not due to the treatment itself, but rather to the interaction with the provider.
    • The acupuncturist assures the patient that he knows how to relieve the back pain, and he provides a complicated explanation with all kinds of impressive, esoteric oriental terminology.
    • He explains that his system is derived from ancient Chinese wisdom and that his needles will adjust the flow of qi through the patient’s meridians to restore health.
    • He takes the patient into a quiet back room, has him lie down and relax, and spends half an hour or more doing up-close-and-personal hands-on treatment.
    • He is charismatic, shows great interest in the patient, asks a lot of questions, and may uncover another unrelated problem that needs treatment.
    • After treatment, he prompts “You feel better now, don’t you?” and the patient feels a social pressure to agree.
    • Instead of dismissing the patient with a prescription, he asks him to return over and over, maybe 3 times a week for several weeks; and when the initial course of treatment is finished, he may want the patient to keep seeing him for treatments to maintain health and prevent future problems. He develops a strong, caring doctor-patient relationship.
The plain needle insertion has been given the Cinderella treatment and transformed into an enhanced package of suggestion, expectation, relaxation, ancillary psychological effects, personal interactions, etc. Acupuncture treatment is ready to go to the ball and wow the prince. It is the ideal placebo package; it’s hard to imagine how anything could be devised that would be better at eliciting placebo responses.
So acupuncture, with no specific effects but many nonspecific treatment effects, will appear to outperform a standard treatment that offers some small specific effects but little in the way of nonspecific enhancements.
The common argument is that it doesn’t matter how a treatment succeeds: it should be used because patients report feeling better faster than with standard care, that it is more effective in a practical sense. But fooling patients with nonsense about imaginary meridians, qi, and acupuncture points amounts to lying. Using placebos and offering fantastic explanations undermine the doctor-patient relationship; and this kind of thing leads people to think non-critically, to accept other kinds of pseudoscience, and to reject science-based treatments that might help them objectively or even save their life in the future.
Cinderella treatments: beautified and dressed up with added enhancements. Why not add as many as possible of these Cinderella enhancements to standard, science-based treatments? Without lying or misrepresenting our knowledge? These pragmatic trials don’t show that acupuncture works; they show that the way standard treatments are offered provides fewer nonspecific effects and could stand a bit of a makeover. That should be our goal.

   Does popularity lead to unreliability in scientific research?
David Gorski
Link
One of the major themes here on the Science-Based Medicine (SBM) blog has been about one major shortcoming of the more commonly used evidence-based medicine paradigm (EBM) that has been in ascendance as the preferred method of evaluating clinical evidence. Specifically, as Kim Atwood (1, 2, 3, 4, 5, 6, 7, 8) has pointed out before, EBM values clinical studies above all else and devalues plausibility based on well-established basic science as one of the “lower” forms of evidence. While this sounds quite reasonable on the surface (after all, what we as physicians really want to know is whether a treatment works better than a placebo or not), it ignores one very important problem with clinical trials, namely that prior scientific probability matters. Indeed, four years ago, John Ioannidis made a bit of a splash with a paper published in JAMA entitled Contradicted and Initially Stronger Effects in Highly Cited Clinical Research and, more provocatively in PLoS Medicine, Why Most Published Research Findings Are Wrong. In his study, he examined a panel of highly cited clinical trials and determined that the results of many of them were not replicated and validated in subsequent studies. His conclusion was that a significant proportion, perhaps most, of the results of clinical trials turn out not to be true after further replication and that the likelihood of such incorrect results increases with increasing improbability of the hypothesis being tested.
Not surprisingly, CAM advocates piled onto these studies as “evidence” that clinical research is hopelessly flawed and biased, but that is not the correct interpretation. Basically, as Steve Novella and, especially, Alex Tabarrok pointed out, prior probability is critical. What Ioannidis’ research shows is that clinical trials examining highly improbable hypotheses are far more likely to produce false positive results than clinical trials examining hypotheses with a stronger basis in science. Of course, estimating prior probability can be tricky based on science. After all, if we could tell beforehand which modalities would work and which didn’t we wouldn’t need to do clinical trials, but there are modalities for which we can estimate the prior probability as being very close to zero. Not surprisingly (at least to readers of this blog), these modalities tend to be “alternative medicine” modalities. Indeed, the purest test of this phenomenon is homeopathy, which is nothing more than pure placebo, mainly because it is water. Of course, another principle that applies to clinical trials is that smaller, more preliminary studies often yield seemingly positive results that fail to hold up with repetition in larger, more rigorously designed randomized, double-blind clinical trials.
Last week, a paper was published in PLoS ONE Thomas by Thomas Pfeiffer at Harvard University and Robert Hoffmann at MIT that brings up another factor that may affect the reliability of research. Oddly enough, it is somewhat counterintuitive. Specifically, Pfeiffer and Hoffmann’s study was entitled Large-Scale Assessment of the Effect of Popularity on the Reliability of Research. In other words, the hypothesis being tested is whether the reliability of findings published in the scientific literature decreases with the popularity of a research field. Although this phenomenon is hypothesized based on theoretical reasoning, Pfeiffer and Hoffmann claim to present the first empirical evidence to support this hypothesis.
Why might more popular fields produce less reliable research? Pfeiffer and Hoffmann set up the problem in the introduction. I’m going to quote fairly generously, because they not only confirm the importance of prior probability, but put the problem into context:
Even if conducted at best possible practice, scientific research is never entirely free of errors. When testing scientific hypotheses, statistical errors inevitably lead to false findings. Results from scientific studies may occasionally support a hypothesis that is actually not true, or may fail to provide evidence for a true hypothesis. The probability at which a hypothesis is true after a certain result has been obtained (posterior probability) depends on the probabilities at which these two types of errors arise. Therefore, error probabilities, such as p-values, traditionally play a predominant role for evaluating and publishing research findings. The posterior probability of a hypothesis, however, also depends on its prior probability. Positive findings on unlikely hypotheses are more likely false positives than positive findings on likely hypotheses. Thus, not only high error rates, but also low priors of the tested hypotheses increase the frequency of false findings in the scientific literature [1], [2].
In this context, a high popularity of research topics has been argued to have a detrimental effect on the reliability of published research findings [2]. Two distinctive mechanisms have been suggested: First, in highly competitive fields there might be stronger incentives to “manufacture” positive results by, for example, modifying data or statistical tests until formal statistical significance is obtained [2]. This leads to inflated error rates for individual findings: actual error probabilities are larger than those given in the publications. We refer to this mechanism as “inflated error effect”. The second effect results from multiple independent testing of the same hypotheses by competing research groups. The more often a hypothesis is tested, the more likely a positive result is obtained and published even if the hypothesis is false. Multiple independent testing increases the fraction of false hypotheses among those hypotheses that are supported by at least one positive result. Thereby it distorts the overall picture of evidence. We refer to this mechanism as “multiple testing effect”. Putting it simple, this effect means that in hot research fields one can expect to find some positive finding for almost any claim, while this is not the case in research fields with little competition [1], [2].
The potential presence of these two effects has raised concerns about the reliability of published findings in those research fields that are characterized by error-prone tests, low priors of tested hypotheses and considerable competition. It is therefore important to analyze empirical data to quantify how strong the predicted effects actually influence scientific research.
In other words, new scientific fields are almost, by definition, full of hypotheses with low prior probabilities, because, well, it’s new science and scientists don’t have a sufficient base of research to know which hypotheses are likely to be true and which are not. I particularly like this phrase, though: “low priors of tested hypotheses and considerable competition.” Sound familiar? That’s exactly what’s going on in CAM research these days, the difference being that most CAM hypotheses are unlikely ever to become more plausible. In any case, as more and more groups start doing CAM research (multiple independent testing) and more and more literature is published on CAM, given the very low prior probability of so many CAM modalities, it is not at all surprising that there are, in essence, papers reporting positive findings in the scientific literature for pretty much any CAM claim, and we can expect the problem to get worse, at least if this hypothesis is true.
Is it true,though? Pfeiffer and Hoffmann chose a rather interesting way to test whether increasing popularity of a field results in the lower reliability of published findings in that field. They chose to look at published reports of interactions between proteins in yeast (S. cerevisiae, a common yeast used in the lab to study protein interactions, often using a system known as the yeast two-hybrid screen). The interesting aspect of this choice is that it’s not clinical research; it is far more likely to find definitive yes-no answers and validate them than it ever is in clinical research.
What Pfeiffer and Hoffmann did first was to use the text mining system iHOP to identify published interactions between proteins and genes in titles and abstracts from the PubMed database and expert-curated data from IntAct and DIP, major databases for protein interactions consisting of expert-curated interactions extracted from the scientific literature and high throughput experiments, as an additional source for published statements on protein interactions. Overall, there were 60,000 published statements examining 30,000 unique interactions. The reported frequency of each interaction was then compared to data derived from recent high throughput experiments based on yeast-two-hybrid experiments, high-throughput mass spectroscopy, tandem affinity purification, and an approach that combines mass-spectroscopy and affinity purification. The strength of this approach is that these high throughput results are not influenced by popularity because they make no a priori assumptions about the protein interactions studied and look at thousands of interactions nearly simultaneously; thus, they are not influenced by either effect being examined, the inflated error effect or multiple testing effect. The weakness of this approach is that these high throughput methods are not free from errors or bias and they do not test all the interactions that can be found in the literature. Finally, Pfeiffer and Hoffmann estimated the popularity of various proteins, or the corresponding genes that encode them, by estimating the number of times the protein or its gene is mentioned in the scientific literature, comparing the popularity of the protein interaction partners and how often the high throughput analysis confirmed interactions reported in the literature.
The results are summarized in this graph:
(Click to go to original, larger image.)
Figure 1A shows what I consider to be a bit of a “well, duh” finding. Specifically, what it shows is that, the more frequently a specific protein interaction is reported in the literature, the more likely it is to be correlate withthe results found in the high throughput experiments, or, as the authors put it:
Interactions that are described frequently in the literature tend to be confirmed more frequently.
This is reassuring in that it at least implies that the more frequently a result is reported by independent labs, the more likely it is to be correct. After all, repetition and confirmation by other investigators are hallmarks of science. However, panels B and C, looking at both datasets examined, show a decreasing reliability for reports of protein interactions that correlates with increasing popularity of individual protein partners in the interactions. In addition, Pfeiffer and Hermann tried to look at the inflated error effect and the multiple testing effect, and found evidence for both being operative, although the evidence for a multiple testing effect was stronger than it was for the inflated error effect–ten times stronger in fact. In other words, if the results of this study are accurate, random chance acting on the testing of hypotheses with a low prior probability of being true is a far more potent force producing false positives than either fraud or “statistical significance seeking.”
So what does this mean for clinical research or, for that matter, all scientific research? First, one needs to be clear that this is a single paper and has looked at correlation. The correlation is strong, and, given the methodology used, does imply causation, although it cannot conclusively demonstrate it. What it does imply, at least from the perspective of clinical trials research, is that the more investigators doing more research on highly improbable hypotheses, the more false positives we can expect to see. Indeed, I’d love to see Pfeiffer and Hermann look at this very question.
My impression–and, let me emphasize, it is just that, an impression–is that the number of seemingly “positive” CAM studies is increasing. Certainly, it’s not unreasonable to hypothesize this to be the case, given the growth of NCCAM over the last decade, to the point where it is funding approximately $120 million a year of little but studies of low probability hypotheses. Add to that the well-known problems of publication bias and the so-called “file drawer” effect, in which positive studies are far more likely to be published and negative studies far more likely to be left in the “file drawer,” and the problem can only get worse. On the other hand, remember that this study looks only at the reliability of individual studies, not the reliability of the literature on topics in totality. The more times a result is seen in science, in general the more reliable that result will be. Similarly, the more frequently a positive result is seen in well-designed clinical studies, the more likely it is to be true. That’s why it’s so important to look at the totality of the scientific literature; the problem is, of course, winnowing out the huge number of crap studies from that totality. All Pfeiffer and Hermann’s work does, at least for me, is to provide evidence supporting a phenomenon that most researchers intuitively suspect to be true.
Whether that’s confirmation bias or real evidence that it is true will await further study.
REFERENCES:
Ioannidis, J. (2005). Why Most Published Research Findings Are False PLoS Medicine, 2 (8) DOI: 10.1371/journal.pmed.0020124.
Pfeiffer, T., & Hoffmann, R. (2009). Large-Scale Assessment of the Effect of Popularity on the Reliability of Research PLoS ONE, 4 (6) DOI: 10.1371/journal.pone.0005996

The perils and pitfalls of “patient-driven” clinical research
David Gorski on July 30, 2012
Link
Dying of cancer can be a horrible way to go, but as a cancer specialist I sometimes forget that there are diseases that are equally, if not more, horrible. One that always comes to mind is amyotropic lateral sclerosis (ALS), more commonly known as Lou Gehrig’s disease. It is a motor neuron disease whose clinical course is characterized by progressive weakness, muscle atrophy and spasticity, with ultimate progression to respiratory muscles leading to difficulty breathing and speaking (dysarthria) and to the muscles controlling swallowing. The rate of clinical course is variable, often beginning with muscle twitching in an arm or a leg or slurring of speech. Ultimately, however, ALS progresses to the loss of ability to move, speak, eat, or breathe. The most common cause of death is from respiratory failure, usually within three to five years after diagnosis, although there is the occasional outlier with a less malignant form of the disease with a slower course of progression who can live a long time, such as Steven Hawking.
In other words, ALS is a lot like cancer in some ways. It is a progressive, fatal disease that usually kills within a few years at most. On the other hand, it is different from cancer in that, at least for many cancers we actually do have effective treatments that prolong life, in some cases indefinitely. In contrast the most effective treatment we currently have for ALS is a drug (riluzole) that is not particularly effective—it prolongs life by months—and can be best described as better than nothing, but not by a whole lot. So it is not surprising that ALS patients, like cancer patients, become desperate and willing to try anything. This is completely understandable, but sometimes this desperation leads to activities that are far more likely to do harm than good. I was reminded of this when I came across a post in the antivaccine propaganda blog, Age of Autism, referring to an article in The Scientist entitled Medical Mavericks. The fortuitous posting of this story, which was apparently designed to try to show that it’s not as crazy as critics have said to be treating autistic children with “Miracle Mineral Solution” (MMS) (which is a bleach) given that the introduction explicitly mentioned Kerri Rivera and the patient described in the article used sodium chlorite to treat his ALS, provided me the opening to discuss a group whose existence and advocacy brings up a complex tangle of issues that boil down to questions of how far patient autonomy should be allowed to go. I’m referring to a company, PatientsLikeMe, which describes itself thusly:
PatientsLikeMe was co-founded in 2004 by three MIT engineers: brothers Benjamin and James Heywood and longtime friend Jeff Cole. Five years earlier, their brother and friend Stephen Heywood was diagnosed with ALS (Lou Gehrig’s disease) at the age of 29. The Heywood family soon began searching the world over for ideas that would extend and improve Stephen’s life. Inspired by Stephen’s experiences, the co-founders and team conceptualized and built a health data-sharing platform that we believe can transform the way patients manage their own conditions, change the way industry conducts research and improve patient care.
The first thing that needs to be kept in mind is that PatientsLikeMe is a for-profit company. Consequently, it’s selling a product, and that product is its analyses of the information patients who sign up with the service reveal about their experiences with various medicine and products. It is true that the company has an admirable list of core values, such as putting patients first, promoting transparency, fostering openness, and creating “wow” (i.e., information that “wows” you), but the company is still selling a platform to its partners and using that platform to attract investors. It is described as a platform “where patients can share and learn from real-world, outcome-based health data.” In essence, PatientsLikeMe is a social media company for patients that mines the reported experience of its patients as, if you believe the hype, thousands of “N of 1” clinical trials.
What I see PatientsLikeMe as doing is trying to mine a very unreliable source, namely thousands of testimonials.
Bleach for ALS?
I first learned about PatientsLikeMe four or five years ago, in part because in my pre-SBM blogging life I had written a series of posts about self-experimentation by cancer patients with an unproven, unapproved drug, dichloroacetate (DCA), a saga I later summarized for SBM two years ago. As regular readers might recall, DCA is a small molecule drug that was used to treat congenital lactic acidosis in children through its inhibition of the enzyme pyruvate dehydrogenase kinase. This inhibition shifts the metabolism of glucose towards oxidative metabolism in the mitochondria and away from glycolysis, the product of which is lactic acid. In January 2007, Dr. Evangelos Michelakis of the University of Alberta had the clever idea of using DCA to target the Warburg effect in tumors and published an article in Cancer Cell describing preclinical experiments in rats in which DCA resulted in marked shrinkage of multiple tumor types. The Warburg effect was first described in 1928 by Otto Warburg and refers to the tendency of many tumors to rely on glycolysis for their energy supply rather than oxidative phosphorylation, even in the presence of oxygen, which in the case of normal cells favors oxidative metabolism. It turns out that DCA showed some promise against glioblastoma in a phase I clinical trial, but it is a long way from being approved.
Because DCA is a small molecule and relatively easy to synthesize, a man named Jim Tassano decided that he would sell “pet DCA” to treat cancer in pets. Tassano fooled no one, of course; people were not buying DCA to treat their dogs. His real purpose was to take advantage of desperate cancer patients, some of whom flocked to his BuyDCA.com website to purchase what he claimed to be pharmaceutical grade DCA. It also led to what I saw as a most disturbing phenomenon, namely self-experimentation by cancer patients with DCA and discussion among patients on online forums, which was even likened to “clinical trials,” complete with reports of success based on magical thinking more than anything else. You will see echoes of this same story and the issue it raises in the article referenced by AoA.
This brings us to the link between MMS and PatientsLikeMe. MMS, as you recall, is being touted as a “miracle cure” for conditions as disparate as cancer and autism. It is nothing more than a 28% solution of sodium chlorite, a chemical widely used for water purification. A couple of months ago at that yearly quackfest of the “autism biomed” movement, Autism One, Terri Rivera gave a talk in which she advocated using MMS to treat autism. Her method of delivery included oral, bathing in it, and per rectum, as in MMS enemas. Yes, we’re talking bleach enemas to treat autism. Administered orally according to Rivera’s protocol, the amount of chlorine dioxide (the active chemical liberated when sodium chlorite is mixed into aqueous solution) is more than 120 times the amount that a child could expect to consume drinking tap water.
It turns out that the story being touted by AoA is about a man named Eric Valor who is taking sodium chlorite to treat his ALS:
In June 2010, he learned of an experimental ALS drug called NP001, being developed by Palo Alto–based Neuraltus Pharmaceuticals. Then just beginning Phase I trials, the drug targets ALS patients’ overactive immune cells, attempting to reduce the chronic neuroinflammation associated with the disease. It seemed promising to Valor, but unfortunately, his disease was too advanced for him to qualify for the trial. “I made various attempts to get [enrolled], but failed,” says Valor, who responded to The Scientist via e-mail because his ventilator limits his speech.
If he couldn’t participate in the trial, maybe he could get the drug another way, Valor reasoned. After an exhaustive literature search on PubMed, he identified the drug’s “cruder” precursor as WF10, which is available for purchase abroad. But he quickly learned that importing it from Thailand would cost more than $12,000 for a year’s supply. He then set out to see if he could get his hands on what he suspected—based on a literature search and a 2006 patent tracked down by fellow ALS patient and friend Rob Tison—was the active ingredient of NP001: sodium chlorite, which is used in water-purification kits for campers and in municipal water treatment.
This is an interesting story on many levels, not the least of which is the apparent finding that sodium chlorite might have an actual therapeutic use. But what is NP001? Due to its proprietary nature, it is not straightforward to figure out just what NP001 is from online sources. For instance, I found what was purported to be the structure of NP001 at the ALS Therapy Development Institute. Notice that there’s no chlorine atom anywhere in chemical structure, making it incredibly unlikely that this molecule would generate chlorite as its active compound. However, more searching around the web made me question whether this is the actual structure of NP001, starting with this press release from the company describing promising results for the drug in a phase I study. Unfortunately, the trial appears not to have been published yet (PubMed searches using the usual suspects, such as “NP001,” “Neuraltus Pharmaceuticals,” and the study’s principal investigator Robert G. Miller, among other strategies, failed to turn up anything); so I went to patent searches and found that Neuraltus Pharmaceuticals has patents for various buffered release systems for sodium chlorite, for example, this one, although searching using the term “NP001” didn’t turn up anything related to ALS. Elsewhere, it is stated that NP001 is a modified form of WF10, which is basically sodium chlorite and has been tested with varying results as an immune modulator for various conditions. In brief, WF10 is a stabilized chlorite matrix that appears to have immunosuppressive effects, although it also appears to stimulate natural killer cell cytotoxicity and decrease macrophage activity. This patent describes the claims for WF10.
To sum it all up, the best that I can figure out is that NP001 appears to be some form of sodium chlorite in a matrix designed to control its release, perhaps with another compound. Certainly, this FDA application for orphan drug status by Neuraltus Pharmaceuticals, Inc. for sodium chlorite pretty much cemented in my mind that that’s what NP001 is, as did the information on this discussion forum, which described how WF10 and NP001 relate to the drug from Thailand referenced in The Scientist article. Moreover, NP001 was granted not only orphan drug status but fast track status as well.
Now here’s the problem with the sort of “do it yourself”-style “N of 1” clinical trials advocated by PatientsLikeMe:
Last June, Valor began taking oral doses of the chemical, which is not approved for the treatment of any disease. And he thinks it’s working. “I have improved breathing, which makes transfers [off his ventilator] much more comfortable,” Valor says. “My voice is louder and speech somewhat clearer. I am able to flex muscles that were previously still (though not enough for useful movement).”
So, basically Valor is taking MMS for his ALS. We don’t know the concentration or the dose that he’s taking. The key phrase above is that Valor thinks it’s working; we have no idea whether there is any objectively measured improvement in his motor function. There are well-established rating scales for severity of ALS symptoms, and it would be much more useful to know whether these patients have any measurable improvement in concrete, objective measures of motor neuron and muscle function. I’ve looked, and I haven’t been able to find any. Valor’s impression could be the result of expectation bias, confusing correlation with causation (i.e., waxing and waning of symptoms correlating with drug administration by random chance alone), or other cognitive quirks to which humans are prone that lead to a mistaken impression of causation.
In addition, in The Scientist article, there is the story of Ben Harris, an ALS patient who was enrolled in the phase II clinical trial of NP001 last year:
At first, Valor didn’t tell other ALS patients about his experimentation, hoping to first establish that it was safe, “but by September, it had leaked,” he says. Medical physicist and ALS patient Ben Harris, who was enrolled in the NP001 trial, was the one to get the ball rolling. NP001 had improved his strength, speech, breathing, and ability to swallow, and he wanted to keep taking the drug after his participation in the trial ended. “The more I learned about [sodium chlorite], the more excited I got, and I felt it was too important to keep a secret,” says Harris, who also responded to questions via e-mail due to limitations associated with his disease. So he started a discussion thread on the website of Cambridge, Massachusetts-based biotech ALS Therapy Development Institute (ALS TDI), sparking widespread interest that helped launch the do-it-yourself (DIY) trial. Now more than two dozen other ALS patients are taking oral sodium chlorite, and recording their results on the social networking site PatientsLikeMe.
Here is Harris’ phase II trial report, as posted on PatientsLikeMe, along with others. Note that most of the patient evaluations for NP001 or sodium chlorite can only be viewed if you have a PatientsLikeMe account. However, the two reports that are publicly available list the drug as having “major effectiveness,” and have a lot of subjective accounts of symptomatic improvement with precious few objective measures (one, actually). At one point, Harris reports a large increase in the grip strength of his hands (the only objective measurement), but by the end of his participation in the trial in November 2011 his impression was this:
I just received my very last infusion of NP001. I think overall if I have declined during this study period it has been at a small fraction of the rate before starting the study. The only thing I can say that is worse is that I am more frequently wiping my lips. I think my lip muscles have atrophied but my tongue has not, neither have my left arm or leg. My left arm was getting weaker just before I started the study but it has held strong since. Along with the decrease in strength on my lips has been a very slight decline in my speech. I have been recording my speech weekly since August and it is very difficult to detect a change so the difference is very small. I believe overall NP001 has drastically slowed my progression if it hasn’t stopped it. Overall, I am better than I was when I started the study.
However, elsewhere, he wrote:
The one way I think I may have declined is in drooling. I seem to have to wipe my lips more often than before and I noticed night time drooling for the first time about one month after starting the study. But the difference is very very small compared to what it was like about 9 months ago. I think perhaps my tongue is stronger but my lips are a little weaker. It is difficult to decide whether I am improving or declining, which in itself is a good thing. Before starting the study I would notice differences in my state every month and they were obvious to me. Right now I am struggling to decide whether there are changes in my state from 4 months ago.
Another thing that you should note: There is a 50-50 chance that Mr. Harris received the placebo. He even acknowledges this elsewhere by telling a reporter that he always tells patients to whom he speaks of this possibility. Yet he wrote his PatientsLikeMe diary as though he knew he was receiving the experimental drug. He seemed to assume that he was in the experimental group and, when the NP001 trial was over, decided to do what Eric Valor has been doing and start taking sodium hypochlorite on his own, as described in this Wall Street Journal story back in April:
Although no patient knows whether or not he received the drug or a placebo, Mr. Harris says he experienced dramatic improvement in his ability to swallow after his first infusion of NP001. After he finished with the infusions on the Neuraltus trial, he wanted to continue. He worried that swallowing sodium chlorite wouldn’t be effective because it would be neutralized in the stomach, so in January, he started injecting it every three weeks. He says he feels no side effects. Mr. Harris says he didn’t inform Neuraltus about what he is doing, but feels that it shouldn’t have bearing on the formal trial since he didn’t start infusing himself until after the NP001 infusions were completed.
Mr. Harris isn’t sure he is benefiting but said last week that he gave himself an infusion and by evening, his swallowing had improved. The real proof, he says, will come when the participants do an analysis of the data collected online. Mr. Harris says he knows that the results will never compare to what Neuraltus is doing.
Eventually, the blinding will be lifted, and Mr. Harris will know. However, he could well be mistaken when he says that his use of sodium chlorite after his treatment as part of the trial was done has no bearing on the trial. If there are late measurements of motor neuron function and survival (and I assume there are), his use of what he believes to be NP001 could skew the trial data.
Unfortunately, this PatientsLikeMe “trial” is not a proper clinical trial at all, and it is highly unlikely that the “analysis” of the patient reports and data collected online will show any sort of conclusive evidence of benefit or lack thereof. The reason, of course, is that this is no more than a collection of anecdotes without a control group, without a standardized protocol covering dosage and administration of the drug, and without even a standardized source of the drug. That’s not to say that analyzing the data might not be useful for examining patients’ perceived side effects and for comparing perceived benefit to actual benefit measured objectively, but in reality what PatientsLikeMe is doing is more likely than not pretty close to useless for determining actual efficacy.
A previous PatientsLikeMe ALS “trial”
The Scientist article mentions a previous PatientsLikeMe trial, which, as it turns out, was what brought PatientsLikeMe to my attention in the first place. For example, here is a story from 2008 about ALS patients trying lithium to treat their ALS based on one promising study.
Just as Tassano sparked “wild experimentation” of patients using DCA to treat cancer with little or no medical supervision based on a single study using rat tumor models, patient experimentation with lithium to treat ALS appears to have been sparked by an Italian study that reported a significant slowing of the progression of ALS. This study looked at both a mouse model and reported the results of a small randomized trial of lithium plus riluzole versus riluzole alone. As a result of this study, ALS patients are taking lithium off-label:
Felzer began taking lithium in January, and his scientifically minded family reached out to other ALS patients. “All those people are taking it anyway,” said Alan Felzer, whose smile remains bright and his gaze sharp even as the rest of his body fails him. “So it only made sense to keep track of what was happening.”
The task of leading the ALS-lithium project fell to Felzer’s daughter, Karen, a U.S. Geological Survey earthquake researcher. Her partner in the effort was Humberto Macedo, a 42-year-old computer systems analyst, father of six and ALS patient in Brasilia, Brazil.
The study grew naturally out of the strong reliance of ALS patients on one another for information, Macedo said.
Working online, Karen Felzer and Macedo recruited nearly 200 patients worldwide to take a specific lithium dosage and answer standard surveys to gauge their symptoms. They began running their study through a Web site called PatientsLikeMe.com, using it to attract volunteers and track their progress.
However well-intentioned Humberto Macedo and Karen Felzer, the organizers of this effort, may have been, they clearly didn’t understand why unrandomized, uncontrolled trials, particularly trials with no objective outcome measurements, are so prone to bias and false positives. After all, what they did was nothing more than the collection of anecdotes, and, as I like to say, the plural of “anecdote” is not data. Granted, that’s a bit of an overstatement in some cases, but not by much. The reason is that this survey didn’t even rise to the level of a case series, given that it relied only on Internet reporting rather than the actual direct assessment of the patient’s condition by a health care professional. That didn’t stop PatientsLikeMe from proclaiming:
Together, with all the patients involved, we will run the first real-time, real-world, open and non-blinded, patient-driven trial. We believe we will have the power, within months, to begin answering the question of how much lithium modifies the progression of ALS. Unlike a blind placebo control trial, we are watching the use of this drug in the real world, and because of the number of patients and our system’s sophisticated data modeling, we can determine the significance of each reported change in each patient as he/she deviates from his/her predicted course. There are many risks to our approach, patient optimism, the placebo effect, uncertain quality, and many other variables will compromise our data. Despite these, and many other challenges, we remain committed to solving this problem.
This “problem,” as it was described, probably cannot be solved because it is inherent in the very design of this study, which was neither randomized nor blinded. There were no valid controls, only in essence historical controls (i.e., the “predicted course”). What’s even more puzzling is that the organizers of this trial even seemed to recognize these problems, acknowledging that the placebo effect, patient optimism, and many other variables may compromise the data. Yet they nonetheless expressed optimism that their “sophisticated data modeling” could overcome these problems. How? They never said. They didn’t seem to realize, either, that what they were doing is not new. There’s a reason why randomized, double-blind clinical trials using either placebo or the standard of care as the control have come to be viewed as the strongest form of clinical evidence for the efficacy of a treatment, and it’s because non-blinded and non-randomized trials are prone to all sorts of biases. They just don’t give answers reliably unless the superiority of one therapy is so much greater than that of the control that it’s undeniable and detectable after just a few patients treated.
So what’s the harm? This is, of course, not an unreasonable question for patients faced with a fatal disease like metastatic cancer or ALS. As hard as it is to believe, even for patients with ALS it is possible to make things worse. Lithium, for instance, is not a benign drug. It can cause neurological complications and diabetes insipidus. It also requires monitoring to achieve therapeutic serum lithium levels. In the case of DCA for cancer, the drug could cause neuropathy and other complications. Overall, when taken as a whole, it is far more likely that patients will be harmed by taking experimental or off-label drugs than significantly helped. Worse, in the “real world” case of most drugs, with effects that are not as dramatic as an antibiotic curing a urinary tract infection, it’s difficult, if not impossible, using methodology like that of PatientsLikeMe.com to detect an actual therapeutic effect
As it turns out, lithium doesn’t improve survival or neurologic function in patients with ALS. Ironically, PatientsLikeMe published its observations several months before the official phase IIb clinical trial in a Nature Biotechnology paper and trumpeted on its blog that it had “refuted a published clinical trial,” specifically, the original clinical trial in PNAS that suggested that lithium might be useful in ALS, stating:
PatientsLikeMe developed a novel algorithm designed to match patients who reported taking lithium with a number of other ALS patients that had similar disease courses. By using a matched control group, PatientsLikeMe was able to reduce biases associated with evaluating the effects of treatments in open label, real world situations and improve the statistical power of the study making each patients contribution more meaningful.
Maybe. More likely, it was nothing more than wishful thinking to believe that this novel algorithm, whatever it is, does an adequate job of controlling for the enormous number and quantity of biases that exist in anecdotal reports from patients. Indeed, I can’t help but think that it is a rather apt comparison when AoA equates PatientsLikeMe “to many of the autism biomed boards.” The main difference is that in the case of PatientsLikeMe there is a company asking members of its website to take polls about the effectiveness of whatever they are trying at the moment and then trying to analyze the anecdotes in order to sell the results to its business partners. In fact, when you come right down to it, PatientsLikeMe is a lot like another social networking site, namely Facebook. Like Facebook PatientsLikeMe has members communicating online. Also like Facebook, its business model is to sell the results of its analyses of the data these members provide. Facebook sells to advertisers; PatientsLikeMe sells to its business partners. In all fairness, to be sure, PatientsLikeMe appears to be a lot more transparent than Facebook. It also doesn’t sell its members’ personal information to advertisers and marketers. Even so, its business model is not that different from that of Facebook. Certainly, there is a lot of marketing going on.
Patient autonomy versus clinical trials, PatientsLikeMe-style
ALS is a horrible disease. I know I’ve already said that, but it is something that bears repeating. Consequently, I can’t blame a single patient for wanting to get his hands on NP001, even to the point of being willing to mix up some home-brewed sodium chlorite or buy non-pharmaceutical grade chemical from whatever supplier he can find. I get it. They don’t have the time to wait for drugs to wend their painfully slow way through the FDA approval process any more than patients with advanced cancer do. Like Derek Lowe, I can’t even say that I wouldn’t be sorely tempted to do the same thing myself, particularly with chemicals that are small molecules that can readily be purchased elsewhere outside the context of a clinical trial. I can even somewhat understand the motivation behind PatientsLikeMe. No doubt the founders of the company think they are somehow accelerating drug discovery while doing good. Unfortunately, they have yet to show that they are accelerating anything. If you look at their publications, none of them demonstrate positive efficacy, and most of them (for instance, the latest three) are about patient attitudes and validation of ratings scales. Nowhere is there any example of a study that has shown anything about the efficacy of a new treatment that adds anything to what conventional clinical trials show. For example, in the case of the lithium trial, no one’s going to believe the results of and Internet survey (which is what, in essence, the lithium trial was); they’re going to wait for the results of a real clinical trial.
PatientsLikeMe has always promoted a rather radical idea, namely that it is possible to tell whether a drug works by analyzing, in essence, a bunch of testimonials from a group of self-selected patients self-medicating with therapies in an unblinded fashion with no proper control group. I suppose it’s worth seeing if it is possible to glean any useful information out of the morass of stories that constitute the database utilized by PatientsLikeMe, but I’ve never had much confidence that the company would be successful at it, and I’ve seen no publications yet that suggest to me that my original assessment was wrong. From what I’ve seen thus far, from a scientific and drug development standpoint PatientsLikeMe appears to have failed, even as it has been wildly successful as a social network for patients. Unfortunately, with this most recent foray into reporting on home-brewed NP001-lookalike drugs (a.k.a. sodium chlorite), PatientsLikeMe is entering dangerous territory that it has eschewed in the past. Remember DCA? PatientsLikeMe could have done then for DCA what it’s doing now for sodium chlorite, but at the time it did not—and rightly so, I might add. Indeed, from a strictly analytic standpoint, DCA would have been a much better test case because the DCA patients were using was chemically identical to DCA that showed promising results in the original rodent studies. In contrast, we have no idea whether the sodium chlorite patients reporting to PatientsLikeMe are using is chemically identical to NP001. Very likely it is not.
There’s a reason why AoA loves PatientsLikeMe, and that’s because it cannily sees it as a more sophisticated version of what “autism biomed” proponents have been doing in discussion groups for a long time: Setting up an online place where people trying anything under the sun up to and including the rankest form of quackery can get together and swap anecdotes and testimonials. One wonders how long it will be before members of the “autism biomed” movement starts infiltrating PatientsLikeMe in order to see if the company will examine their assessments of the various interventions to which they subject their autistic children, such as MMS enemas, hyperbaric oxygen chambers, untold varieties of supplements, chelation therapy, and other dubious and sometimes dangerous interventions. Indeed, given that PatientsLikeMe has recently published on its assessment of the efficacy of drugs given for off-label indications, one wonders whether its founders would be willing to examine whether Lupron works for autism. The precedent has been set with sodium chlorite, after all.
There will always be a conflict between wanting to do something now for suffering patients, damn the consequences, and following the scientific method to demonstrate efficacy and safety. It’s perfectly understandable, and the need of dying patients is especially compelling. We have seen both extremes. Indeed, until 1906, drug sellers could make essentially any claim and sell essentially anything to the public as a drug without regulation. We all know how well that worked out. Early in the history of the FDA, companies often tested new drugs by sending them to doctors to offer to their patients, asked for little information regarding side effects and complications, and had no standard criteria for efficacy. Efforts of groups like PatientsLikeMe.com may be an improvement over a century ago in that they do go to great efforts to ask patients about side effects and changes in how they feel, but, boiled down to its essence, this patient-driven trial is the same thing as the unscientific experimentation of that era. It benefits neither science nor patients. Unfortunately, the genie is out of the proverbial bottle; we are not likely to be able to put it back in.

   Annals of Questionable Evidence. a new study reveals substantial publication bias in trials of anti-depressants
Kimball Atwood
Link
Part IV of the ongoing Homeopathy series will have to wait a day or two, because it is superceded by a recent, comment-worthy publication. Nevertheless, “H series” fans will find here a bit of grist for that mill, too.
An important role for this blog is to discuss problems of interpreting data from clinical studies. Academic medicine has committed itself, on the whole, to scientific rigor–to the extent that this is possible in messy, clinical (especially human) trials. Several tools have been proposed, and to a varying extent used, to enhance the rigor of clinical research and the reporting of clinical research. One of those tools is the registering of clinical trials prior to recruiting subjects. Registration would stipulate a trial’s a priori hypothesis(es), design, planned endpoints, and planned statistical methods, among other things. This would guard against several problems: publication bias—the tendency for some trials, usually “negative” ones, to go unreported; selective reporting of the results of a trial, if some are pleasing but others are not; and post hoc data analysis—finding data after the fact to suggest a novel hypothesis that will falsely be portrayed as an a priori hypothesis. Publication bias is also known as “selective publication” or the “file drawer problem”; post hoc analysis is also known as “data dredging” or “HARKing” (Hypothesizing After the Results are Known).
An article in the Jan. 17 issue of the New England Journal of Medicine demonstrates the usefulness of a trial registry:
Selective Publication of Antidepressant Trials and Its Influence on Apparent Efficacy
Erick H. Turner, M.D., Annette M. Matthews, M.D., Eftihia Linardatos, B.S., Robert A. Tell, L.C.S.W., and Robert Rosenthal, Ph.D.
For years, the FDA has required Investigational New Drug (IND) applications from the sponsors of trials intended to support FDA-approval to market new drugs or to market old drugs with new indications. The IND application includes the proposed trial’s protocol. When the trial is finished, the sponsor must submit a New Drug Application (NDA) that includes both the raw data from the trial and the investigators’ conclusions. The FDA conducts its own analysis of the data and compares that to the conclusions of the investigators. These applications thus fulfill several of the requirements necessary to accomplish the policing function described above.
In the past, the information that the FDA received was not made public. Recently that has changed, and Turner and co-authors took advantage. They looked at 74 FDA-registered trials of 12 anti-depressant agents that eventually won approval, involving 12,564 subjects, conducted between the years 1987 and 2004. They compared the FDA findings with the published reports of the same trials. What they found was striking: of the 37 studies that the FDA viewed as positive, 36 were published. But of the 35 studies that the FDA viewed as “having negative or questionable results,” 22 were never published and 11 were “published in a way that, in our opinion, conveyed a positive outcome.” All in all, “According to the published literature, it appeared that 94% of the trials conducted were positive. By contrast, the FDA analysis showed that 51% were positive.”
The authors could not tell whether the publication bias was due more to investigators not submitting manuscripts or more to journals rejecting those manuscripts. Other work suggests that the former is the predominant reason. The Turner study was also unable to discern whether industry-sponsors of the trials may have influenced decisions to submit manuscripts. Other evidence, including the infamous Olivieri and Vioxx cases, suggests that this may have been a significant factor. Some research contracts between academic investigators and their industry-sponsors have included “gag clauses” requiring investigators to obtain permission from sponsors before submitting manuscripts.
Non-publication of human research is unethical. The most important reasons are that it exploits human subjects for no purpose, and that by skewing data it may encourage the medical profession and the public to pursue inferior or even dangerous treatments. Many in academic medicine consider it a form of scientific misconduct. Publication bias is only one of many reasons that industry-sponsored research may yield erroneous conclusions. To be fair, another recent study found that this may be less pervasive than some think.
Science and Clinical Research
There are many other pitfalls in clinical research. Among the most important are the nearly universal misuse of “frequentist” statistics and its partner-in-crime, the lack of formal estimates of prior probability. Citing such pitfalls together with publication bias and other biases, a learned author has argued persuasively that “Most Published Research Findings are False.” This by no means supports the Post-Modern conclusion, to which a few readers might be drawn, that nothing can really be “known” in an objective, externally-verifiable way. On the contrary: some published research findings are true, and the scientific weeding process inevitably identifies them. Thus we know many objective, externally verifiable truths about health and disease, most of which we have learned only in the past 150 years or so–probably less than 1% of the time that our species has had enough intelligence to do so.
We know a lot about pulmonary, cardiac, endocrine, gastrointestinal, renal, and neurophysiology. We know that there is an immune system and we have devised ways of inducing it to protect us from many of the plagues that afflicted our less knowledgeable ancestors. We know how to fashion molecules that can interact with physiologic processes; we already have quite a few that are useful, and we will develop many more. We can even predict with some certainty which conditions will be amenable to future pharmacologic interventions (obesity) and which will not (Down’s syndrome). We know exactly what it is that gives someone sickle cell disease, although we haven’t yet figured out how to cure it. We know that the production of a special glycoprotein by the parietal cells in the gastric mucosa is necessary for the absorption of dietary vitamin B-12. We know that the absence of that protein, if undiagnosed and untreated, can lead to anemia, severe neurologic problems, and dementia, and we know how to reverse those sequellae. We know that malaria is caused by certain protozoa and that mosquitoes are their vectors. We know that quinine cures malaria by killing those protozoa, and we know how quinine does this. We know that many of those protozoa are now resistant to quinine, and we know that those resistant strains have flourished because of selection pressure caused by us. We know that AIDS is caused by an RNA virus, we know how it spreads, and we can suppress it to a considerable extent in most patients, even if we can’t yet eradicate it.
In addition to those few, arbitrary examples, we know vastly more about human biology and medicine. We know enough to fill numerous, thick textbooks, and we are learning at breakneck speed–even if we are still, by any reasonable estimate, in the early stages of discovery. The goal for clinical research, then, is to strive for scientific rigor. The author of the article cited above offers several suggestions to “improve the situation,” and we on this blog will be discussing those and more.

   Open-Access Peer Review: Increasing the Noise To Signal Ratio
Val Jones
Link
Readers of Science Based Medicine are quite familiar with the distressingly common logical leap made by disgruntled healthcare consumers into alternative medicine. It goes something like this: I had a terrible experience with a doctor who [ignored/patronized/misdiagnosed] me and I also heard something horrible in the media about a pharmaceutical company’s misbehavior [hiding negative results/overstating efficacy/overcharging for medications], therefore alternative treatments [homeopathy/acupuncture/energy healing, etc.] must be more effective than traditional medicine.
Much to my dismay, a similar logical leap is being made about online health information. It goes something like this: Peer reviewing is biased and often keeps innovative research hidden to the world at large, therefore the best kind of peer review is open-access where anyone in the world can contribute.
You may feel free to slap your forehead now.
While I have absolutely no doubt that doctors have their shortcomings, and that some have created less than pleasant healthcare experiences for their patients – the solution to these shortcomings is not to dive headlong into snake oil. Moreover, I agree that the current peer-review process has its flaws and limitations – the solution is not to ask Aunt Enid in Omaha what she thinks of the recent meta-analysis of perioperative beta blockers in patients having non-cardiac surgery.
Peter Frishauf, the founder of Medscape, recently published a webcast editorial predicting that:
“Peer review as we know it will disappear. Rather than the secretive prepublication review process followed by most publishers today, including Medscape, most peer review will occur transparently, and after publication.”
He goes on to describe a Wikipedia-like review scenario where:
“Any user can start an article, link it to related sources, and publish revisions with a click of the mouse. Anyone who reads an article can edit it.”
I know and like Peter very much, and his foresight (that publishing should become open-access), combined with the leadership of editorial heavy-weight, Dr. George Lundberg, led to the creation of the first really successful, quality, free online medical journal. This was no small feat, and a sure victory for global medical education efforts.
But the reason for The Medscape Journal’s success is not the “democratization” of peer review – but the democratization of access to trustworthy information. The quality controls are still in place – and must remain so – otherwise its value as a peer-reviewed journal will be utterly lost. Who should trust the edits of unqualified readers? Should science be determined by popular vote? Should all research be published by journals, regardless of its fatal flaws?
I wrestle daily with people who truly believe that “the wisdom of the crowds” will revolutionize medical information. These people are part of a “do it yourself” healthcare movement – spurred on by frustration from a broken system, overconfidence in their ability to interpret complex data, and an ignorance of the potential biases, statistical waffling, and methodologic flaws that so easily mislead even the most savvy reviewers. They like to imagine themselves as hip, anti-establishment pioneers – daring to “push the envelope” in the face of paternalistic scientists who cling tenaciously to an unreasonable status quo. In reality, however, they’re working to unravel the progress we’ve made over a century of painstakingly controlled scientific experiments – systematically designed to remove bias, errors, and false assumptions wherever possible and uncover the raw truth of things. As David Gorski (and The Onion) has pointed out, science is hard.
Now let me explain why open-source peer review does not solve the current publishing “bias problem” and why it would, in fact, be inferior to our current system. Advocates of various Web 2.0 tools argue that flinging wide the doors to all readers (à la Wikipedia) is the single best way to clarify, edit, and correct information. I can see how this may be useful for historical documents, but scientific literature is another matter altogether. Not only are unqualified reviewers likely to do more harm than good with their well-intentioned edits, but the fact of the matter is that there is little incentive for most people to do any reviewing at all.
Slate recently discovered (quite accidentally) this timeless truth: people are unlikely to participate in activities that have no personal value to them. Their review of physician-rating sites revealed dismal participation rates. The site creators believed that scads of patients would immediately descend upon their open-source tools in order to selflessly fill out long questionnaires about the quality of their physician interactions. Alas, the “revolutionary” and empowering rating tools that promised to help patients find out the “truth” about their doctors’ quality were met with a yawn and the sound of crickets. The few ratings that did exist (one site had 137,832 listed doctors and 5,709 patient reviews) tended to skew towards negative experiences, completed by patients whose frustration had reached sufficient levels to carry them through the arduous rating process.
And so, since good peer review of the medical literature is far more arduous than even physician ratings, those who choose to participate in an open source scenario are likely to have an axe to grind. The potential for bias online probably exceeds bias in the current peer review process, where reviewers are hand-selected for subject matter expertise from institutions where there is no suspected conflict of interest, and both authors and reviewers are purposefully blinded so that pre-conceived notions of individual or institutional competencies are less likely to play a role. The fact that the reviewer names are never revealed further reduces incentive to “make a name for oneself” through academic posturing of a self-aggrandizing nature.
Suggesting that the current peer review process should be replaced with an open-source free-for-all is kind of like suggesting that double-blind, placebo-controlled trials should be substituted with unblinded trials where subjects and investigators both know if they’re getting a placebo or the treatment under investigation. Wouldn’t it be fairer for everyone to just have open access to all that information? Isn’t it unnecessarily secretive and paternalistic to hide the “truth” from study subjects? Maybe a better way to figure out if a drug really works is to get all the patients together to brainstorm about their experiences? Poppy cock!
Now, I agree that we could improve our current peer review process by vastly expanding the reviewer pool (this would also help to speed up the process of getting good research published and available for public consumption). But that pool must still consist of reviewers with real expertise – and the motivation problem must be solved. What’s in it for the reviewer? Well, why not devise a system whereby CME credit is offered to qualified physicians who wish to participate in peer review? Why not allow online journals to pull from a large, shared pool of carefully vetted but anonymous review volunteers?
There are ways to improve the scientific peer review process, and there are ways to increase the noise to signal ratio. Open-access peer review is certainly an example of the latter.

Zombie Science
Retractions of scientific studies do not always mean that the studies die a deserved death. Sometimes they live on as zombie studies, continuing to be cited by other researchers and having an effect on the scientific discussion. We can fix this.
Steven Novella on October 11, 2017
Link

This article, unfortunately, is not about the science of zombies, but rather science that continues on after it has effectively died. What makes a scientific study dead (and undead)?
The most obvious and definitive way in which a scientific study can be officially pronounced dead is if it is retracted. Scientific research is born into life (to continue this metaphor) when it is published in the peer-reviewed literature. Being retracted from the journal in which it was published is scientific death (or at least it should be).
Retraction watch, which is an excellent resource, cited over 650 retractions in 2016. There has been an overall trend of increasing retractions, up about 10 fold from 2000. The reasons for retraction vary – about 11% are for fabrication or falsification of data, 17% for self-plagiarism, 16% for plagiarism, 28% for honest errors, and 11% for irreproducible results. The majority of retractions come from high-impact journals and come from the biomedical sciences.
Retractions themselves are not necessarily a problem. They represent the peer-reviewed literature policing itself and correcting errors when found. The real question is – what happens to a paper after it is retracted? One way to quantitatively address that question is to track citations. A citation is when one scientific paper references another. This usually means that the paper is building on the previous work in some way. It may be literally reviewing previous studies, or just using it to support its conclusions or premises.
As an aside, one bias in the literature that we have not discussed enough on SBM is citation bias. This is a clearly demonstrated bias for researchers to cite positive studies over negative studies, regardless of study quality. Essentially this is the scientific literature equivalent of confirmation bias.
Citations are also the way journals measure their impact factor. The more their articles are cited by other articles, the higher their impact factor (which relates to how profitable they are). Therefore there is a clear incentive to publish positive studies over negative studies, and in general studies that are more likely to be cited (because they are new and interesting). These are also the studies most likely to be retracted, which may explain why high impact journals have a relatively high retraction rate.
So what happens to citations when an article is retracted? The online version of that article is then watermarked with a big “Retracted” label or a notice at the top indicating the article is now retracted. Print versions that are out there in the world are, of course, unaltered. Online curators of peer-reviewed articles, such as Pubmed and Medline, are also notified. Is this enough? Probably not.
DARPA has been looking into this question. They found that one paper published in 2005 and retracted in 2010 was cited 667 times, about half of those citations after it was retracted. That is a zombie paper, living on in an undead state after its official death by retraction.
The ripple effect of citations is also huge. Not only can a paper be cited many times, but the papers that cite it are also cited. Even if we just consider these second degree citations, a typical paper can have thousands of such connections, and a high impact paper, tens of thousands. That is the nature of modern science – researchers do not usually work in isolation. The literature is a giant web of communication and collaboration.
There are deeper questions that need to be researched. What is the effect of citations to retracted papers? Do they invalidate the results or conclusions of the paper making the citation? This is most likely for systematic reviews and meta-analysis – if one of the papers you relied upon in your review is revealed as bogus, that might dramatically change the results of the review. Other citations might be negligible, merely one of many citations simply to establish some basic premise, and unaltered by a single retraction.
We can think about what should happen in a perfect world. I think this is yet another situation in which we are not optimally leveraging the power of digital information. When a paper is retracted, it should no longer be possible to cite it. If a new paper is submitted that cites a retracted paper, that should be caught in the editorial review. Researchers should be able to easily determine if any paper they are trying to cite is a zombie paper.
Further, once retracted every lead author of every paper that cited the now retracted study should get a message and should be essentially forced to revise their paper to account for the retraction. This may be as simple as removing the citation, but it may also require changing the text if it refers to or depends on the now defunct citation. It may even require reanalyzing data to remove the retracted data from the analysis. The worst case scenario is that a paper which depended upon the now-retracted citation may itself need to be retracted, with the process repeating for any paper that cited it.
In other words, the retraction should have the same ripple effect throughout the web of scientific literature that the original paper had, extricating its tendrils from the web and making all needed corrections. Yes, this is a lot of work, but that is precisely why we need to leverage the power of computers to make it happen. This also sounds like a job for AI software, and is likely a lot less complicated than Google’s search algorithms. Essentially we have the technology to do this if we wanted to.
The bigger picture is that the scientific literature has not completely made the transition from traditional print journals to an online digital existence. That transition needs to be completed, with journals, search engines, and libraries rebuilt from the ground up digitally. How and what gets “published” should be optimized for scientific progress using digital technology, and not tied to the old restrictions. Editors should no longer be able to say things like, “We don’t have the space to publish boring exact replications.” The idea of “space” when it comes to publishing is a paper restriction, not a digital restriction.
Further, the response to retractions should be immediate and absolute. Those zombie papers need to be staked through the heart (OK, that’s vampires, but you get the idea) and permanently laid to rest. It should simply not be possible to have a single citation after being retracted.
Retraction, of course, is not the only problem with published studies or the only way in which a study can become a “zombie.” Some studies simply come to conclusions that we now know are wrong, even though the study itself was legitimate. That should be noted also.
DARPA is looking into ways to label published scientific studies with a rating system that indicates how reliable the results are. Impact factor is not enough, and in fact may be misleading as it is biased towards positive studies that are, if anything, more likely to be retracted. I have suggested previously that preliminary studies should be clearly labeled with a warning that their results are preliminary, exploratory, or speculative only, and should not be cited as reliable evidence. “For use by other researchers only!”
I seriously think a company like Google should be brought in and given a grant to develop a system for monitoring, tracking, and grading studies in a more thorough way that is optimized for scientific utility, progress, and reporting in the media. This is not meant to replace the collective judgement of the scientific community, but to codify it and make it more transparent.
It’s time to build a scientific literature infrastructure for the 21st century.

   Nosodes Redux: “I hate those meeces to pieces!”
Mark Crislip
Link
Life and medicine generate facts and experiences that require conceptual frameworks that aid in understanding. It is no good have a pile of facts if they cannot be understood within a broader understanding.
The practice of Infectious Diseases, while certainly aided by understanding anatomy, physiology, microbiology, chemistry and the other sciences that form the core of medicine (referred to in Medical School as the basic sciences), gains a broader appreciation from the concepts of evolution. Infectious Diseases, at its most fundamental level, is applied evolution, and understanding evolution often adds greater insight into infectious diseases. Me find bug, me kill bug, me go home may be my motto, but it is meant in jest.
There have been papers or books that have added conceptual frameworks to my understanding of the natural world and medicine. Besides evolution, there was Observations on Spiraling Empiricism a classic that all health care providers should read, as it outlines the cognitive errors we all make in prescribing medications; I have discussed this article before.
There is The Drunkard’s Walk: How Randomness Rules Our Lives by Leonard Mlodinow. So often the explanation of why something happens is a shrug of the shoulders; feces occurs. The book formalized my understanding that much of what happens is random and without cause. The challenge in medicine is trying find a pattern in the randomness of life upon which to base a diagnosis. It is equally important to recognize when patterns are not there. All too often what is seen as a pattern is our imposing structure on what are random events. Or maybe that really is a bunny in the clouds. Clinical study results often occur by chance and having a significant ‘P’ value may still be due to randomness if the study is measuring nonsense.
Recently, Why Most Published Research Findings Are False, by John P. A. Ioannidis has added to my understanding of the old saw that half of everything you learn in medical school will be proven false after you graduate, the problem is that they do not tell you which half.
Ioannidis has 6 rules for evaluating studies that stem from his evaluation of the literature:
Corollary 1: The smaller the studies conducted in a scientific field, the less likely the research findings are to be true.
Corollary 2: The smaller the effect sizes in a scientific field, the less likely the research findings are to be true.
Corollary 3: The greater the number and the lesser the selection of tested relationships in a scientific field, the less likely the research findings are to be true.
Corollary 4: The greater the flexibility in designs, definitions, outcomes, and analytical modes in a scientific field, the less likely the research findings are to be true.
Corollary 5: The greater the financial and other interests and prejudices in a scientific field, the less likely the research findings are to be true.
Corollary 6: The hotter a scientific field (with more scientific teams involved), the less likely the research findings are to be true.
It is why much of medicine and science is tentative and emphasizes the importance, as best as can be determined at a given time, of considering all the information. I tell the residents that a big part of becoming a Doctor is learning to be comfortable with incomplete and changing information. Being a specialist is being ignorant with style.
What matters is the totality of the evidence… instead of chasing statistical significance, we should improve our understanding of the range of R values—the pre-study odds—where research efforts operate. Before running an experiment, investigators should consider what they believe the chances are that they are testing a true rather than a non-true relationship.
It is where the Cochrane reviews fail, and fail so spectacularly, on topics as diverse as the efficacy of influenza vaccine to the efficacy of homeopathy. Failure to consider all the information casts doubt on the former and suggests legitimacy of the former. In the world of clinical medicine, when considering an intervention, I have to take into consideration as much of the literature as I can, from basic plausibility to the always messy clinical trials.
What separates a proponent of science based medicine from practitioners of SCAM’s is that those in SBM are aware of the all too numerous foibles and errors to which humans are prone, distorting a true understanding of the natural world. What makes a good skeptic, and a good doctor, is an understanding of the multitudinous ways we fool ourselves and others.
Which brings us back to nosodes and the reference Do homeopathic nosodes protect against infection? An experimental test by WB Jonas.
Just on the basis of the first five corollaries of Ioannidis, the odds are the study is wrong, not to mention that the pretest probability, the R, that this study, if positive, will be will be testing a true relationship is, lets see. Exactly zero. There is zero pretest probability that any positive result of a homeopathic invention represents an effect from the homeopathic nostrum.
Dr. Jonas, the sole author, is a former director of National Center for Complementary and Alternative Medicine, so, while the study was done at Walter Reed Army Institute of Research, Corollary 5 jumps to mind as perhaps more important than where the study is done.
Lets look at the quality of the study.
Groups of 4 to 6 mice (Corollary 1) received either ethanol (70%) or a varying dilutions of pureed leptospira infected mouse lung/liver/spleen in 70% alcohol. Some of the nosodes had measurable proteins in it. The author measured the amount of proteins in the nostrums using the Lowry procedure, a semiquantitative measurement of proteins, which I remember using as a fellow. It is less accurate when there is a host of other organic materials in the mix, as one might expect from a melange of liver, lung and spleen.
For me, ingestion of 70% alcohol leads to amnesia, so how 70% alcohol can ‘remember’ the initial material better than water is not mentioned in the study. Is a study with 70% alcohol meaningful for ‘classic’ water based homeopathy? Are serial dilutions of 70% alcohol applicable to homeopathy as practiced? Stupid questions, since homeopathy has nothing to do with reality. Must. Remember. This. Is. Tooth. Fairy. Science.
Mice received about 16 doses of a nosode or placebo before infection challenge and about 40 doses after the infectious challenge with leptospirosis.
Compared to alcohol (which is not a placebo, since alcohol does have a variety of immunomodulatory effects, one example from where you find others,) over all the nosode group died after 18.6 hours and the control after 13.7 hours with a P of .002. Overall mortality was 75% in the control, 53 % in the nosode. Standard vaccination was, of course, vastly superior to alcohol or nosode, with 100% protection. There was no dose dependent result, which would be a surprise if a real medication were being used, but not, perhaps, an antigen.
If you expose an animal to a pathogen, say Candida, and then shortly thereafter infect the animal with, say, Listeria, the animals that had the prior exposure to Candida will have decreased mortality compared to those that were not exposed. The immune response is primed for subsequent infectious challenge.
One telling aside in the study is that one mixture had no effect, and that was a 1000C nosode prepared by a commercial pharmacy. The author postulates “perhaps handling while in transit or differences in preparation of the commercial nosode affected the 1000C level.” He also worries that the mechanical succussion does not have the same effect as hand succussion, which in the real world is akin to worrying that Tooth Fairies exchanged the real nosode for mere alcohol while he slept. Of course, Tooth Fairies only concern themselves with teeth, so a bad example.
I suspect, having been involved in bench research back in the day isolating and measuring proteins, that the author did a bad job at making his dilutions (but not the commercial pharmacy) and the nosodes were more contaminated with puree of infected mouse innards than he suspected. The result is the typical mild protective effect you get from prior exposure to antigens. Alternatively, it is just the usual random results you see in small studies combined, perhaps, with yet another example of seeing N-Rays.
Overall not impressive and a low quality study and of little applicability to ‘classic’ homeopathy. And, like many studies in the SCAM literature, not repeated with a better methodology. The author states at the end, “Attempts to replicate these experiments under more controlled conditions (eg with complete blinding (!!) and using aerosol methods for more uniform dosing) are underway.” That was 11 years ago. I wonder what happened to the results. Follow up with improved methodologies is not the strong suite of researchers in the SCAM field. Squirrel!
There must be a less expensive and less cruel way to kill off 142 mice. D-Con is cheaper and spring loaded traps are quicker. Given that homeopathy is divorced from reality, this is more needless cruelty to animals than a reasonable scientific study.
My colleagues use the terms like ‘highly unlikely’, or ‘highly implausible’ to describe the precepts of homeopathy. I guess I am not a true skeptic. Some things are, as the world is currently understood, impossible. Perpetual motion machines are impossible, accelerating past the speed of light is impossible. There are few SCAM’s I would classify as impossible out of hand. Stick needles in people, or pop their neck, and you will get some sort of effect, perhaps not the effect intended (infection and stroke). There is always the possibility that the SCAM is doing something, perhaps not in ways envisioned by the practitioner. Even something as goofy as iridology or live blood analysis has the opportunity, albeit very rare, to make a real diagnosis. Only homeopathy has the honor of being the only SCAM based totally on fantasy.
I would classify the basic precepts of succussion and dilution to increase potency as impossible, not implausible. I’m sorry, not dilution, potentiation, or as it is more appropriately called in fantasy fiction, casting a spell. Homeopathy is 100% nonsense, and, like its nostrums, uncontaminated by reality. Sometimes it is argued that in the lower concentrations of spells, there may be a molecule or two of an agent that could have an effect, to which I call shenanigans. I subscribe to the concept of dose-effect of agents and there is a concentration below which molecules will have no effect. The alleged toxins in vaccines are not toxic because the concentrations are inconsequential; the effects of homeopathic nostrums at initial spell castings are as equally unable to have an effect as the later spell castings.
If homeopaths were to say the act of serial dilution and succussion slowly transformed the water into Unicorn tears, a universal curative, it would not make the underlying concepts less rational or divorced from the natural law. I may not be doing a very good job of suppressing my total lack of respect for the underlying fictions of homeopathy.
At one time, as an example, we did not know how aspirin worked, only that it was effective. But the science of chemistry, pharmacology, etc. eventually lead to the understanding that aspirin results in irreversible inactivation of the cyclooxygenase enzyme, which prevents the production of prostaglandin and thromboxane. As you increase the dose of aspirin, the receptors are filled until all the receptors are used up and then there is no more effect by giving more aspirin. Elucidating the action of aspirin did not require a completely new understanding of the basic sciences. If homeopathy did indeed work, then all we know of the natural world is wrong. Which is more likely: all of the basic sciences are correct or homeopathy is correct. And maybe homeopathy does give us a deeper insight in to the natural world, since science as we know it as about to be overturned.
In the end, I wonder at times if SBM and medical journals are the correct forum for discussions of the validity of the underlying precepts of homeopathy. What is the proper forum for the discussion of magic? The fantasy fiction section, not the science section, of the library. Or perhaps homeopathy is the purview of the anthropologist and psychologist as a cultural delusion.
Delusions are irrational beliefs, held with a high level of conviction, that are highly resistant to change even when the delusional person is exposed to forms of proof that contradict the belief. Non-bizarre delusions are considered to be plausible; that is, there is a possibility that what the person believes to be true could actually occur a small proportion of the time. Conversely, bizarre delusions focus on matters that would be impossible in reality. For example, a non-bizarre delusion might be the belief that one’s activities are constantly under observation by federal law enforcement or intelligence agencies, which actually does occur for a small number of people. By contrast, a man who believes” that the fundamental precepts of homeopathy are true “holds a belief that could never come to pass in reality.
In medicine we determine that a behavior is pathologic when it starts to have negative consequences for the patient or others. Alcohol is the classic example where use blends from fun to pathologic as harm accrues in the patient and his environment. Homeopathy would appear to be another. I look forward to the inevitable ‘when you point a finger you have three pointing back at yourself’ reply.
When I was first in practice, I saw a patient in clinic for ‘parasites.’ The room reeked of garlic as he was continually chewing on raw garlic cloves as he said it suppressed the parasite.
In discussing his symptoms he told me that the air was full of these parasites, and they would land on his skin and burrow in.
Probably Delusions of Parasitism, I thought, as no parasite has this life cycle. But it is not unusual for patients to mis-attribute symptoms to an erroneous cause, so I had to make sure he did not have some illness he was mistakenly attributing to parasites.
“How do you know they are parasites?”
“I collect them.” he said, holding up a large brown jar that rattled dryly as he shook it.
“Can I see them? And where do you get them?”
“From my nose,” he replied as he dumped three cups of dried boogers on the exam table.
To my credit I neither screamed nor vomited, although I could not eat garlic for over a year.
I see no conceptual difference between that jar of ‘parasites’ and a jar of homeopathic nostrums, save the former, to my knowledge, never killed a child.

   Science-based medicine, skepticism, and the scientific consensus
David Gorski 
Link
There are some arguments made in blogs, articles, or books that strike me so hard that I remember them, even three and a half years later. Sometimes I even file them away for later use or response if the issue raised by them is interesting, relevant or compelling enough to me. Although this topic is a bit broader than many of the topics I write about for this blog, I think it also goes to the heart of science-based medicine and communicating scientific skepticism about medicine to the masses. A few years back, a Swedish blogger named Martin Rundkvist made a rather provocative observation about skepticism. Specifically, he argued that a “real skeptic always sides with scientific consensus.” Among his reasons was this one:
Science presupposes that all participants have a skeptical frame of mind and arrive at conclusions through rational deliberation. If a large group of knowledgeable people working in this way arrive at a consensus opinion, then there is really no good reason for anybody with less knowledge of the subject to question it. Informed consensus is how scientific truth is established. It’s always provisional and open to reevaluation, but as long as there’s informed consensus, then that’s our best knowledge. Humanity’s best knowledge.
Although at the time I saw where Martin was coming from, I found this viewpoint somewhat disturbing, leading me to echo Martin’s own words in response to his own rhetorical question asking whether accepting a scientific consensus is nothing more than “kowtowing to white-coated authority”: Well, yes and no.
For a skeptic and supporter of science-based medicine, in matters of science it is undoubtedly true that the scientific consensus is always the best place to start when evaluating unfamiliar issues. While it is certainly possible that a given scientific consensus regarding an issue can be wrong in almost any area, it nonetheless almost always represents the best current scientific understanding. It is also correct, as Martin argues, that legitimate authority matters. I emphasize the word “legitimate” because in pseudoscience arguments from authority are common, but rarely is the authority relevant to the point being argued. Often it’s not even legitimate, as in when anti-vaccine activists point to Andrew Wakefield’s work as justification for their claims that vaccines cause autism and other conditions. Like Martin, I’m more inclined to accept the pronouncements of someone who has actually dedicated his or her life to studying the issue systematically; i.e., an expert. If the topic is evolution, then that expert would be an evolutionary biologist. If the topic is the Holocaust, then a historian specializing in World War II and/or the Holocaust would represent an appropriate expert. For cancer, an appropriate expert would be an oncologist (or an surgical oncologist, such as myself). The list goes on.
Where I start to have a bit of a problem with Martin’s viewpoint is when I start to contemplate the nature of scientific consensus itself in many areas of science. Not all scientific consensuses are created equal because, in different fields the strength of scientific consensus can vary quite markedly depending upon the topic or even the subtopic within the topic. For example, the scientific consensus supporting the theory of evolution, particularly common descent, is exceedingly strong. It’s one of the strongest of all scientific consensuses, arguably the strongest. Similarly, the consensus that natural selection is a major driving force behind evolution is very nearly as strong. However, as the discussion devolves into more detailed areas, inevitably the consensus weakens. Eventually, subsidiary areas of a discipline are reached where the consensus is weak or where there is no consensus, such as what the function of “junk DNA” is, whether it is subject to natural selection, and if so how much. (Real evolutionary biologists could probably come up with a better example.) These sorts of questions are often at the cutting edge of scientific knowledge, and it is not always easy to recognize what they are. It is also these issues at the edge of our knowledge that are attacked as proxies for the much more strongly supported core theory. Creationists are notorious for this sort of tactic, often using new findings of molecular biology that appear peculiar or out of sync with specifics of evolution as wrecking balls to try to demolish the edifice of “Darwinism.” I can’t count the number of times I’ve seen supporters of “intelligent design” creationism point to either “junk DNA” or the findings of new genomic techniques as “evidence” that “Darwinism” is doomed. Never mind that some of the very strongest support for the theory of evolution, in particular common descent, comes from genetics and that new genomic techniques only keep confirming that.
The same is true of scientific consensuses in many other disciplines, including my own disciplines of medicine and surgery. However, scientific consensuses in medicine are almost never as strong as the theory of evolution. Germ theory of infectious disease probably comes close, but otherwise I’m hard-pressed to think of one. Usually, the strength of a consensus in medicine is proportional to the ratio of data supporting it that comes from randomized clinical trials to data from epidemiological studies, the latter of which are more prone to confounding factors. That does not, however, mean that there doesn’t exist strong consensus about many medical issues. For example, there is, for all intents and purposes, no doubt that HIV is the causative agent for AIDS, the claims of HIV/AIDS denialists notwithstanding. Similarly, there is in essence no doubt that smoking cigarettes vastly increases a person’s risk of lung cancer and heart disease, along with a host of other medical problems. Not even the tobacco companies try to argue against that anymore. When we come to subsidiary questions, however, the consensus is generally not as strong. For example, it has become increasingly appreciated that secondhand tobacco smoke increases the risk of heart disease and lung cancer in people chronically exposed to it. However, because the effect is considerably smaller than it is for people who actually smoke cigarettes, there is a lot more “noise” in the studies, giving more wiggle room for people who dislike the idea of the government banning smoking to claim that such bans are not scientifically supported, and it’s taken a long time for scientific and clinical studies to firm up the conclusion enough to the point that it is now a strong consensus.
Not surprisingly, given the difficulty doing controlled experiments and the nature of the material, which makes it more easily politicized or influenced by biases, a truly strong consensus is harder to come by in the humanities and social sciences. However, even so, it is not impossible. For example, one of my areas of interest is the Holocaust. There is overwhelming and incontrovertible evidence that the Nazi regime embarked on a systematic program to round up and exterminate the Jews in territories they controlled. Methods used included shooting, hanging, gas chambers, and a system of camps designed to literally work their inmates to death through a combination of grueling labor, insufficient food, and unsanitary and crowded conditions. However, this consensus becomes less clear when various issues surrounding the Holocaust are discussed. For example, there is the whole “intentionalism” versus “functionalism” debate. In a nutshell, this debate is over the question of whether the intent was there from the very beginning of the Nazi regime or even before the Nazis rose to power to exterminate the Jews or whether the Holocaust evolved “organically” or “functionally” from the persecution of Jews that began early in the Nazi regime and became increasingly radical and violent in the pre-war era, the radicalization of Nazis carrying out the program, and the question of what to do with the millions of Jews that suddenly fell under Nazi control after Hitler had invaded the Soviet Union. Holocaust deniers love to misrepresent and misuse this debate to claim that Hitler didn’t know about and didn’t order the Holocaust or to claim that there was never a plan to systematically exterminate European Jewry.
One thing that I’ve come to realize is that it’s not always obvious when skepticism devolves into denialism. In some cases it’s pretty easy, for example creationism, Holocaust denial, HIV/AIDS denialism, anti-vaccine activism and the germ theory denialism that often fuels it, and many sectarian belief systems that lead to the use of unscientific “alternative” medicine, such as homeopathy. Sometimes, appropriate skepticism about most scientific issues can exist side-by-side in a single person with credulity and pseudoskepticism about other scientific issues. For example, among skeptics who count themselves science-based, there is a disturbingly large contingent who do not accept the science behind anthropogenic global climate change, as Penn Jillette, for example, does not. Only recently does he appear to be grudgingly dragged towards accepting the scientific consensus in this area. His resistance to the idea appears to be due to his Libertarian beliefs.
One example that I used to use routinely before she shut down her blog was Sandy Szwarc at Junkfood Science. Most of her non-obesity-related posts were often actually not too bad. However, whenever she blogged about diet and obesity, there was usually a problem, sometimes a big problem like the one Steve Novella blogged about a couple of years ago. And it’s not the sort of thing that necessarily jumps right at you off of her blog, as it does from an obvious crank blog like Age of Autism. Certainly it didn’t for me. Rather, it’s the sort of thing you have to read her blog closely for a while (which I did) to start to realize. As I read her blog, more and more it bothered me that all of her “skepticism” was inevitably in the direction that being obese is not only not unhealthy but is actually at least as healthy as not being obese, that eating fatty foods is perfectly fine, and that virtually any study she looks at that says that eating fatty foods or too many calories predisposes to health problems is a pile of crap while any pile of crap study claiming otherwise is the latest and greatest. All of this led her to conclude that virtually every warning made by scientists and physicians about diet is fearmongering. Worse, she had a distressing tendency to use unscientific tactics, such as cherry picking data, attacking consensus, and alleging conspiracies. I also found it telling that, unlike most bloggers, myself included, Szwarc does not permit comments. If there’s one thing that skeptics usually encourage (as we do here on SBM), it’s spirited debate. That’s impossible in a blog that doesn’t permit comments.
In the end, I remain a bit uncomfortable with Martin’s blanket statement that “a real skeptic always sides with scientific consensus.” The reason is that what the scientific consensus actually says is not always that clear for many issues, even among those who work in the field. Indeed, there are comparatively few issues in science (evolution, for example) for which a strong consensus exists, and even fewer in the social sciences and history (the Holocaust, for example) for which an equally strong consensus exists. These are very strong consensuses, and to overturn them would require extraordinary evidence, evidence at least equal to the evidence supporting them. Consequently, when someone says that evolution is false or that the Holocaust didn’t happen (or the lessor form of Holocaust denial, that nowhere near 6 million Jews died), it’s fairly easy to recognize such person as a crank and denialist, and I usually do not hesitate to label them as such when appropriate.
But what about consensuses that are strong but not as bullet-proof, usually because, although there is a consensus, there are fairly wide error bars around the predictions or uncertainty regarding the importance of various factors? The prototypical example of this is anthropogenic global warming, for which there is a strong consensus among climate scientists but still a fair amount of uncertainty about the outcome. Another example, of course, is the scientific consensus about the link between obesity and adverse health outcomes. How do we differentiate legitimate skepticism about the consensus from denialism?
This is where I tend to agree with Mark Hoofnagle. It’s more about tactics and how evidence is used to support an argument. Scientific skepticism looks at the totality of evidence and evaluates each piece of it for its quality. Cranks are very selective about the data they choose to present, often vastly overselling its quality and vastly exaggerating flaws in current theory, in turn vastly overestimating their own knowledge of a subject and underestimating that of experts. In medicine in particular, denialists frequently emphasize anecdotes over epidemiology, clinical trials, and science. They also tend to leap to confuse correlation with causation. A great example that I just saw a week or two ago comes from our “friends” at the anti-vaccine group the International Council on Vaccination, a group that Mark Crislip and I had such fun deconstructing. There, Sherri Tenpenny posted an article entitled Vaccines and Blue Foot Syndrome (crossposted on her own blog). In the post, after citing anecdotes about “vaccine injury,” Tenpenny then segues into a story of a child who dropped a large frozen turkey on his foot, which became painful and blue within hours. The parents are then confronted with doctors who tell her that this is “Blue Foot Syndrome” and that they have shown by epidemiology that it is not linked to being struck by a frozen turkey. It’s an analogy obviously designed to mock scientists who quite correctly refer to the studies that have failed to find a link between vaccines and autism that ignores the fact that correlation does not necessarily equal causation and that the correlations reported by parents are not nearly as obvious as a blue foot after having a frozen turkey land on it. Anti-vaccine activists think they are, though.
In addition, because the mainstream rejects their ideas, there is often a strong sense of being underappreciated—persecuted, even—among cranks, leading them to view their failure to persuade the mainstream of the correctness of their views as being due to conspiracies or money. Global warming denialists, for example, tend to see the currently existing scientific consensus as being politically motivated by the desire of “liberals” to tell them how to live. Evolution deniers view evolution as the result of atheistic scientists wanting to deny God. People like Sandy Szwarc view the consensus that obesity leads to health problems as being due more to moralizing and bigotry against the obese, which, whether it is true or not, is an easy claim to make because there has been and is a lot of bigotry against the obese. Antivaccinationists view the rejection of their belief that mercury in vaccines or even vaccines themselves cause autism by mainstream medicine as evidence that we’re all in the pocket of big pharma. Practitioners of “complementary and alternative medicine” view the rejection of their beliefs in homeopathy, “energy healing,” and vitalism similarly: Science-based doctors must be in the pocket of big pharma! Or they’re close-minded! Or they’re threatened by the “success” of alternative medicine!
What a lot of this distinction boils down to is that crankery, denialism, pseudoskepticism, or whatever you want to call it tends, either intentionally through ideology or unintentionally through an ignorance of the scientific method, to conflate and/or confuse emotiona, nonscientific, and/or ideological arguments with scientific arguments. This is not to say that scientists and skeptics and supporters of SBM are free from their own biases, whether ideological or simply a desired result that they hope to find. Far from it. However, skepticism means applying the scientific method to claims, whatever its faults, scientific method is the best method thus far devised to minimize these biases. As scientists, the reason we use the scientific method is not because we consider ourselves superior to the cranks, but rather because we recognize that we are human too and thus just as prone to falling into the same traps as they. As Richard Feynman once famously said, “The first principle is that you must not fool yourself—and you are the easiest person to fool. So you have to be very careful about that. After you’ve not fooled yourself, it’s easy not to fool other scientists. You just have to be honest in a conventional way after that.” The scientific method is, above all, a methodology by which scientists try to avoid fooling themselves. Skeptics cross the line dividing skepticism and denialism and quacks the line between science and quackery when they forget that.
Moreover, skeptics and supporters of SBM know that science is a work in progress. What is considered correct today may well be modified tomorrow. This change, however, is not brought about by cranks cherry-picking data but by rather skeptical scientists probing for weak spots in our current understanding, making hypotheses, and then testing whether current theory or the new hypotheses make the better prediction. Thus, being skeptical of the consensus is not the mark of the crank. It’s how and why that skepticism exists that distinguishes crankery from genuine scientific skepticism.

   Do Antidepressants Work? The Effect of Publication Bias
Steven Novella 
Link
A recent meta-analysis of the most commonly prescribed antidepressant drugs raises some very important questions for science-based medicine. The study: Initial Severity and Antidepressant Benefits: A Meta-Analysis of Data Submitted to the Food and Drug Administration, was conducted by Irving Kirsch and colleagues, who reviewed clinical trials of six antidepressants (fluoxetine, venlafaxine, nefazodone, paroxetine, sertraline, and citalopram). They looked at all studies submitted to the FDA prior to approval, whether published or unpublished. They found:
Drug–placebo differences in antidepressant efficacy increase as a function of baseline severity, but are relatively small even for severely depressed patients. The relationship between initial severity and antidepressant efficacy is attributable to decreased responsiveness to placebo among very severely depressed patients, rather than to increased responsiveness to medication.
The press has largely reported this study as showing that “antidepressants don’t work” but the full story is more complex. This analysis certainly has important implications for how we should view the body of evidence for these antidepressants. It also illuminates the possible role of publication bias in the body of scientific literature – something that has far ranging implications for science-based medicine.
The Strengths and Weaknesses of this study
The primary strength of this analysis is that it looked at all clinical trials for these antidepressants submitted to the FDA prior to approval – whether or not the studies were published. The authors did this specifically to eliminate the effect of publication bias on the data. In pooling the data together using meta-analysis statistical methods the study also able to have much more power than a single study (the analysis included 5,133 total subjects). The study design also allowed for comparison of antidepressant efficacy for differing initial severity of depression – an important question for determining who should and should not be treated with antidepressants.
The study has numerous weaknesses, however. Because the study only looked at pre-approval clinical trials it did not account for all available data. Also, once a drug is approved study designs are more variable as they are no longer specifically designed to meet the criteria for FDA approval and may be more relevant to clinical practice. The analysis only considered a single measure of depression (Hamilton Rating Scale of Depression – HAM-D), and it is possible that other measures of depression may have yielded a different result. The study also included data mainly for severe depression, with only one study of moderate depression and now studies of mild depression. Finally it should be noted that meta-analyses in general are not highly predictive of the outcomes of later large definitive trials. The process of performing a meta-analysis itself has the potential to introduce bias and error.
What does this study mean?
This, of course, if the ultimate question for clinicians and the public – how should the results of this meta-analysis be incorporated into the current practice of medicine? This is a complex question, and can only be answered in the context of all available evidence on the use of antidepressants in the treatment of depression. Not all studies and not all systematic reviews and meta-analyses of the use of antidepressants in depression give us the same results. For example, this recent review found:
Paroxetine yielded consistently and significantly better remission (rate difference [RD]: 10% [95% CI = 6 to 14]), clinical response (RD: 17% [95% CI = 7 to 27]), and symptom reduction (effect size: 0.2 [95% CI = 0.1 to 0.3]) than placebo.
This review looked at multiple outcome measures, and pre and post approval trials, but only considered published studies. Therefore it had different strengths and weaknesses than the current review, and further study is needed to determine how best to interpret these conflicting results.
At this time it is premature to conclude that modern antidepressant medications do not work. There is sufficient evidence for efficacy to continue to use medication as part of the overall treatment approach to depression. The current consensus is that therapy is also a critical component of the long term treatment of depression, and therefore looking at the use of medications in isolation may not reflect their actual clinical use. Multiple studies have now shown that combination treatment (medications and therapy) are better than either alone. There is also evidence that medication treatment is more successful when multiple agents are tried in order to find the optimal treatment. These so-called “real world” treatments are not well reflected in the pre-approval trials considered in this analysis.
But this analysis does indicate that perhaps clinicians should consider alternatives to medication for patients with mild to moderate depression, and reserve medication for more severe patients or those who do not respond to non-medication modalities alone. It also emphasizes (together with other evidence) the need for combination therapy in many patients.
It is also important to note that depression is only one of many clinical uses for antidepressant medications. Other uses include anxiety, panic disorder, neuropathic pain, and eating disorders. Each indication should be considered on its own merits.
The role of publication bias
This meta-analysis is important for what it tells us about the role of publication bias, a term that refers to the tendency for researchers to be more likely to submit, and editors to publish, papers with positive results than with negative results. Therefore any review of the published literature is not a reflection of all evidence, but of a subset of the evidence biased toward positive outcomes. With regard to clinical trials of antidepressants, the pharmaceutical company sponsors of those studies had to report them to the FDA (and therefore they were accessible for this review) but they tended to publish only the positive studies. This is also not the only review to find such a bias. As my colleague, Kimball Atwood, has previously discussed, Turner et. al. looked at trials submitted to the FDA for 12 antidepressants and found:
According to the published literature, it appeared that 94% of the trials conducted were positive. By contrast, the FDA analysis showed that 51% were positive.
We cannot determine whether the bias observed resulted from a failure to submit manuscripts on the part of authors and sponsors, from decisions by journal editors and reviewers not to publish, or both. Selective reporting of clinical trial results may have adverse consequences for researchers, study participants, health care professionals, and patients.
This effect is also not limited to clinical trials of antidepressants, or even to pharmaceutical trials, but plagues all published medical research. Publication bias distorts the body of evidence upon which science-based medicine is dependent. It is a particular problem for systematic reviews and meta-analysis. It even has the potential for creating the impression that a clinical effect exists where it does not.
One way to compensate for the effect of publication bias is to rely more on large, definitive, high profile clinical trials. Such trials are usually the result of a consensus that emerges from a body of literature, with both proponents and skeptics agreeing on the study design, taking into account the results of previous weaker studies. Such trials tend to high profile, meaning that the results cannot be hidden away once the study is concluded. Large individual trials are therefore not subject to publication bias.
But we need to develop further strategies to minimize the effect of publication bias on the literature. One strategy would be to require that all trials involving human subjects be registered in a central database. The results of all human trials could therefore be accessible to researchers, whether or not the results were ever published. Such a system already exists for FDA drug trials, but should be expanded to include all human trials. Also, researchers should be encouraged to submit studies for publication, even when they are negative. And journal editors should make efforts to publish negative results, and not favor positive results that are more likely to grab headlines.

   H. Pylori, Plausibility, and Greek Tragedy: the Quirky Case of Dr. John Lykoudis
Kimball Atwood
Link
Mark Crislip is on vacation, but through an arduous series of shakings and succussions (beating his head against the wall?) we have channeled part of his essence: This post mostly concerns itself with infectious diseases, thanks to several recent posts on SBM that discussed the plausibility of health claims† and that touched on the recent discovery that most peptic ulcer disease (PUD) is caused by a bacterium, Helicobacter pylori. Several comments and statements quoted in those posts reveal recurrent questions regarding both plausibility itself and the history of the H. pylori hypothesis. In this post I’ll attempt to answer some of those questions, but I’ll also insert some new confusion.
Plausibility ≠ Knowing the Mechanism
Let’s first dispense with a simple misunderstanding: We, by which I mean We Supreme Arbiters of Plausibility (We SAPs) here at SBM, do not require knowing the mechanism of some putative effect in order to deem it plausible. This seems so obvious that it ought not be necessary to repeat it over and over again, and yet the topic can’t be broached without some nebbishy South Park do-gooder chanting a litany of “just because you don’t know how it works doesn’t mean it can’t work,” as if that were a compelling or even relevant rebuttal. Let’s get this straight once and for all: IT ISN’T.
Steve Novella explained why at the Yale conference and again here. We talked about it at TAM7 last summer. For a particularly annoying example, read the three paragraphs beginning with “Mr. Gagnier’s understanding of biological plausibility” here.
OK, I’ll admit that I’m beginning to learn something from such frustration. Perhaps we’ve not been so good at explaining what we mean by plausibility. The point is not that we don’t know a particular mechanism for homeopathy, for example; the point is that any proposed mechanism would necessarily violate scientific principles that rest on far more solid ground than any number of equivocal, bias-and-error-prone clinical trials could hope to overturn. The same is true for “energy medicine” and for claims based on non-existent anatomical structures (iridology, reflexology, auricular acupuncture, meridians, chiropractic “subluxations”), non-existent physiologic functions (“craniosacral rhythms“), or non-existent anatomic-physiologic relations (“neurocranial restructuring,” “detoxification” with coffee enemas, dissolving tumors with orally administered pancreatic enzymes). The spectrum of implausible health claims euphemistically dubbed “CAM” is full of such nonsense.
Reader daedalus2u proposed a useful way to clarify the point:
I think the idea of prior plausibility should actually be reframed into one of a lack of prior implausibility. It isn’t that one should have reasons to positively think that something is plausible before testing it, but rather that one should not be able to come up with reasons (actually data) why it is fatally implausible.
Some of what We deem implausible will not be fatally so, of course. Implausibility can be based not only on established physical and biological knowledge, but also on studies, as is the case for sticking needles into people, injecting them with chelating agents, or claiming that autism is caused by childhood immunizations.
Plausibility and History
A second theme, though not as simple, concerns the historical role of plausibility. Reader anoopbal introduced the point:
am not sure if you can apply biological plausibility to every situation. It is usually considered as a weak criterion because it is limited by our knowledge.
If we used plausibility 300 years back, nobody would have used citrus fruits for scurvy nor people would have believed mosquitoes are linked with black water fever.
I think that daedalus’s “reframing” proposal deals with that objection to a large extent. I also don’t think that anoopbal’s examples are all that revealing. It seems to me that empiricism was the main source, other than myth, for plausibility at the time. Plausibility in the biomedical sense is not something that can be usefully discussed for the period prior to about the mid-19th century, when enough was finally known about biology and chemistry to hatch science-based medicine in its full form. Prior to that, most useful diagnostic and treatment methods had been discovered empirically (accidentally).
This is not to say that someone living before the mid-19th century could not have applied plausibility to a medical question—obviously that could happen at any time—but that to attempt to do so, when so much was still mysterious (how cowpox pus worked, microbiology, Avagadro’s number, energy flux in living organisms, physiology, pharmacology, etc.) or ‘explained’ by magic (the Vital Force, miasmas, sympathetic magic, the 4 humors, etc.) would have meant very little by today’s standards. And I do believe that there is a fundamental difference—not merely a foolish conceit about modernity—between what we know today and what we knew 300 years ago. Thus I don’t think that biological plausibility is a weak criterion now, even if it was then.
To give anoopbal his or her due, he seemed to partially agree when he later noted:
And that‘s exactly the limitation of biological plausibility. It is limited by what we currently know. Centuries back our knowledge about earth was limited, and you can’t blame them for believing the earth [was] a flat disc.
On to H. Pylori
Daedalus offered another interesting take on plausibility:
An idea does not have low prior plausibility if it does not agree with prior explanations, it has low prior plausibility if it does not agree with prior data.
Many (most?) scientists make this confusion too. That is because they are thinking on the level of the explanations, not on the level of the data that led to those explanations. The explanations may be wrong, the data that led to them is not.
Given the presumption that the data are accurate, we would all probably agree with this. Daedalus, however, then got a little tripped up:
The idea of using antibiotics to treat ulcers was incompatible with the idea that ulcers were due to too much acid. It was not incompatible with any of the data surrounding ulcer treatment.
So did Harriet Hall:
The idea of treating ulcers with antibiotics was not incompatible with any of the data about ulcers; it was only incompatible with the idea that ulcers were caused by too much acid.
At the time that Barry Marshall and Robin Warren proposed their bacterial hypothesis, there were data suggesting that ulcers were caused by too much acid: acid neutralization or suppression of acid formation resulted in better than 90% healing of peptic ulcers, compared with about 30% for placebo. If such therapies were discontinued after healing, the ulcers typically relapsed, only to be healed again by renewed acid suppression. This did not rule out the possibility of some other factor also being involved, of course, but it would seem to have come pretty close to throwing down the gauntlet of Ockham’s Razor.
Why not just Treat with Antibiotics?
What might it have taken prior to 1984, short of what was subsequently done, to convince the world that peptic ulcer disease could be effectively treated with antibacterial agents? A reader sent just such a question to Steve Novella:
What would Science Based Medicine do if H. pylori was not known, but a study showed that antibiotics given to patients with stomach ulcers eliminated symptoms? I assume that SBM wouldn’t dismiss it outright saying that it couldn’t possibly be helping because antibiotics don’t reduce stomach acid. I assume a SBM approach would do further studies trying to discover why antibiotics work. But, in the meantime, would a SBM practitioner refuse to give antibiotics to patients because he doesn’t have a scientific explanation as to why it works?
A straightforward answer is as follows. Although the question may raise an interesting general point about plausibility, the example is not a good one. Antibiotics are not one medicine but many. They all have side effects, some quite serious. Bacteria are also not one species but many; they have widely differing sensitivities to various antibiotics. Which antibiotic(s) would the study have used, and on what basis? Responsible MDs would not have accepted such a scheme for PUD, because they would have needed to know what they were treating and how to treat it (H. pylori turns out to require three different antibiotics given simultaneously).
A predictable rejoinder to this is that many physicians routinely treat upper respiratory tract infections, most commonly caused by viruses, with an antibiotic. Without going into detail, let me assure you that this does not refute my point: in many cases MDs should not be treating these URIs with antibiotics, and in cases where it makes a bit of sense to do so it is done with a single, short-term antibiotic with a benign risk/benefit profile, known to be effective against the most common community-acquired bacterial culprits of the respiratory tract. This is quite different from attempting to treat a mysterious bacterium that might not even exist, for a disease that already has effective treatments that are safer and have fewer side effects than antibiotics.
Mikerattlesnake got the point:
I think it’s a wise addendum to directly address the logical misstep in the question you received. Those who understand SBM would get the answer from the broad approach taken in your post, but those people aren’t the ones likely to parrot the fallacy.
To put it simply: finding that an antibiotic was effective against an ulcer would indicate a bacterial cause for ulcers that would warrant further study. The reason for that has entirely to do with prior plausibility. Antibiotics are known to fight bacteria. If an antibiotic cures ulcers, it gives us a plausible answer for the mechanism causing ulcers. The questioner makes the mistake of assuming that we would never abandon the assumed cause of ulcers, but SBM looks for mechanisms of action for ailments as well as cures.
So did BillyJoe, with the appropriate caveat:
I like it.
Only one thing though: this could never have happened. As I understand it, the treatment involves taking three different tablets – two antibiotics and an acid suppressing drug – twice a day for a week. How likely is that to have happened by chance?
Discoveries Require Context
Some might assume that Robin Warren and Barry Marshall were the first to discover bacteria apparently living in the human stomach and duodenum, and the first to propose that the bacteria might be involved in diseases of these tissues, but this isn’t the case. Such bacteria were first observed in the 19th century. Over subsequent decades there were sporadic reports of similar bacteria, but they were not necessarily associated with diseases and their presence could not be reliably reproduced. The table of contents of Helicobacter Pioneers: Firsthand Accounts from the Scientists who Discovered Helicobacters 1892 – 1982, edited by Barry Marshall, gives a hint of just how close some investigators came to the truth:
     1.  Helicobacters were discovered in Italy in 1892: An episode in the scientific life of an eclectic pathlogist, Giulio Bizzozero. Natale Figura and Laura Bianciardi
     2.  The discovery of Helicobacter pylori in Japan. Yoshihiro Fukuda, Tadashi Shimoyama, Takahashi Shimoyana and Barry J Marshall
     3.  An early study of human stomach bacteria. A. Stone Freedberg
     4.  Gastric urease in ulcer patients in the 1940′s: The Irish connection. Humphrey J O’Connor and Colm A O’Morian
     5.  How it was discovered in Belgium and the USA (1955 -1976) that gastric urease was caused by a bacterial infection. Charles S Lieber
     6.  A personal history of giving birth to the cohort phenomenon of peptic ulcer disease. Amnon Sonnenberg
     7.  John Lykoudis: The general practitioner in Greece who in 1958 discovered the etiology and a treatment of peptic ulcer disease. Basil Rigas and Efstathios D Papavassiliou
     8.  How I discovered helicobacters in Boston in 1967. Susumu Ito
     9.  How we discovered in China in 1972 that antibiotics cure peptic ulcer. Shu-Dong Xiao, Yao Shi and Wen-Zheng Liu
     10.  Helicobacter pylori was discovered in Russia in 1974. Igor A Morozov
     11.  The discovery of Helicobacter pylori in England in the 1970′s. Howard W Steer
     12.  We grew the first Helicobacter and didn’t even know it!. Adrian Lee, Michael Phillips and Jani O’Rourke
     13.  The Dallas experience with acute Helicobacter pylori infection. Walter L Peterson, William Harford and Barry J Marshall
     14.  The discovery of Helicobacter pylori in Perth, Western Australia. J Robin Warren
     15.  The discovery of Helicobacter pylori, a spiral bacterium, caused peptic ulcer disease. Barry J Marshall
     16.  Helicobacter pylori treatment in the past and in the 21st Century. Peter Unge
Prior to Marshall and Warren, human gastric bacteria were not only inconsistently seen, but were never cultured and hence never characterized in a useful way (for a more basic treatment of this topic, please see my 2004 essay in Skeptical Inquirer). At least two distinct technological advances were necessary to set the stage for the discovery and characterization of H. pylori in humans: first, a simple and safe method for obtaining gastric mucosa specimens from live patients had to be devised; second, the field of bacteriology had to appreciate the existence of highly fastidious organisms and devise methods for growing them in culture. The first of these requirements was satisfied only by the late 1970s, when flexible, fiberoptic endoscopy became widely available.
I am not enough of an historian of bacteriology to state, with certainty, when H. pylori might have first been cultured, if only its existence had been fully appreciated, but it is doubtful that it could have occurred much sooner than it did. Helicobacter Pioneers reports that the first successful culture of any helicobacter species—isolated from mice—occurred in 1968. Other examples of fastidious bacteria have also been characterized relatively recently: mycoplasma pneumoniae and chlamydophila pneumoniae, two organisms that cause atypical pneumonia, were still thought to be viruses until the 1960s; several new species of helicobacter and campylobacter, some of which are human pathogens, have been discovered only in the last 10-15 years.
The culture requirements of H. pylori, moreover, are esoteric: it grows best in an atmosphere of 5% oxygen and is helped by the presence of certain antibiotics to discourage overgrowth by more hardy contaminants, which are almost impossible to avoid when collecting specimens from the stomach via the mouth. Helicobacter takes much longer to grow than most bacteria, and but for serendipity Warren and Marshall might have missed it. They abandoned their first 34 culture attempts (or, more precisely, “junior microbiology staff” abandoned them) in spite of multiple variations of media and temperatures, after no growth had occurred within 48 hours. It was only after a five-day Easter vacation, during which the 35th attempt was left undisturbed, that tiny, transparent colonies appeared.
John Lykoudis: the Real Galileo of PUD?
An intriguing story in Helicobacter Pioneers is found in chapter 7: John Lykoudis: The general practitioner in Greece who in 1958 discovered the etiology and a treatment of peptic ulcer disease. You can read most of this chapter at the Google Books website. If it is accurate, it makes the answer to the question that the reader posed to Dr. Novella not so straightforward as Mikerattlesnake, BillyJoe, and I argued above: it inserts the “new confusion” that I promised at the beginning of this post. Lykoudis’s story has all the necessary tragic elements:
a general practitioner in a small, isolated town in Greece, prompted by a single clinical observation, developed on his own the concept that PUD and gastritis had an infectious etiology. As if this was not enough, this most unlikely student of PUD proceeded to devise an apparently effective treatment, based on the antibiotics of his time.
Lykoudis’s treatment, apparently developed by trial and error, consisted of 3 antibiotics (2 quinolines and streptomycin, for you microbiology/infectious disease enthusiasts out there) and vitamin A, taken orally. He patented this regimen in a pill that he named Elgaco, “from the Greek word for ulcer (= elkos), gastritis and colitis” (for which he also asserted that his treatment was effective). He eventually claimed to have treated 30,000 patients with nearly perfect results and no toxicity. According to the authors of the chapter,
The success of Elgaco cannot be quantified from extant notes on thousands of patients, because the outcome of each patient is not recorded. We have concluded, however, that his treatment was successful, based on the following considerations. First, our current understanding of the etiology and treatment of PUD makes it plausible that his treatment was effective. Second, there is the written testimony (some of it sworn, as explained later) of many of the patients who were treated by Lykoudis. All report prompt responses to his therapy. In some cases, patients even detail that radiographically proven ulcers were cured following treatment with Elgaco and that such cure was confirmed by repeat radiological series. Third, Lykoudis had a large following and despite fierce opposition from the establishment, patients flocked to him from all over Greece.
In spite of this, his attempts to make his discovery known to the world were rebuffed at every turn:
He encounter[ed] formidable obstacles in convincing the medical establishment, the Greek regulatory authorities and the pharmaceutical industry. In fact, Lykoudis spent the rest of his life engaged in incessant activity to propagate his treatment of PUD and gastritis. His archives, some made recently available by his family, make it clear that he was fully aware of the importance of his discoveries. They also convey an almost suffocating sense of frustration…
[He was] completely shunned by the medical establishment of his time, or at best, considered an eccentric provincial physician…
In 1966, Lykoudis attempted to publish his observations in the Journal of the American Medical Association, but his manuscript entitled “Ulcer of the Stomach and Duodenum” was rejected…Unfortunately, no copy of this manuscript survives for re-evaluation in the light of current knowledge.
Lykoudis did, however, publish his own booklet, “The Truth about Gastric and Duodenal Ulcer.” In it he wrote:
There is no doubt that gastritis and duodenitis, which have gastric and duodenal ulcer as their complication, are inflammations due to an infectious agent…
Lykoudis made numerous attempts to get his remedy approved by the Greek Drug and Pharmacies Administration, to no avail. He even managed to enlist the aid of influential politicians:
In 1967, Lykoudis succeeded in getting the attention of the Prime Minister’s office. His correspondence with the Minister of Health on 21 August, 1967, a sad document indeed, is revealing. He registers his frustration that medications with apparently no effect on PUD were approved, whereas Elgaco was repeatedly rejected. He proposes, in essence, a phase III trial: 100 PUD patients to be treated at a State hospital by the eminent professors, 50 with conventional treatment and 50 with Elgaco. ‘Their refusal to approve it is understandable, but their refusal to test it is not!’ he writes.
Lykoudis continued:
If the study proves them correct, they will be vindicated and I will become a laughing stock…It is dramatically urgent to clarify this issue…Too much, endless talking, which leads nowhere, while it is simple to resolve this in a practical way. Only facts constitute the truth.
Yet again he was refused. Lykoudis also tried, unsuccessfully, to interest several drug companies in his regimen. The final insults were these:
…he was referred for disciplinary action to the Athens Medical Association, of which he was a member, ‘because (a) he prepared and distributed an unapproved medicinal preparation…and (b) he made his method publicly known to attract patients’…On 6 November 1968…the Disciplinary Committee, presided over by a neurology professor, fined him 4000 drachmas…
A more serious problem for Lykoudis was his indictment in the Greek Courts…
In the latter instance numerous former patients came to his support; one of them testified that Lykoudis “treated also many poor ulcer patients free of charge.” We are not told the outcome of the indictment.
Lykoudis died in 1980 without knowing that he would soon be vindicated. His story is disturbing because it is an almost perfect hybrid of two entirely different possibilities: on the one hand, a legitimate innovator who is unfairly rejected and persecuted, in spite of heroic efforts over more than 2 decades to prove his theory; on the other, a classical example of unwitting foolishness, bordering upon quackery or sociopathy.
It is only in hindsight that we can grant that there is a good chance that it wasn’t the latter. Consider the striking parallels, however, to Nicholas Gonzalez, whose main arguments have consisted of patient testimonials and case reports selected by himself, who claims that his regimen is nontoxic, who claims to treat some patients for free, who was hounded by regulatory boards for a time, who found political allies to help defend him, and who for years pleaded that all he wanted was a chance to test his regimen:
I believe in research. I don’t want this to be out there until we prove it works by the strictest standards of orthodox medicine. What I have wanted from the day I began researching this under Dr. Goode at Cornell in 1981, was to do appropriate clinical trials.
Again: Discoveries Require Context
My sense, reading the story of John Lykoudis, is that he was treated unfairly, and I think that most people would agree. A major caveat is that the authors of the chapter are clearly sympathetic to him, and it’s quite possible that another account would read differently.
Whether fair or not, it seems to me that the major weakness in Lykoudis’s case is that he never characterized the putative bacteria in any way: he didn’t see them, he didn’t provide direct evidence of them for others to examine, and he didn’t culture them.
Even his sympathetic biographers recognized this. Although they attributed his failure to “his lack of academic credentials” and even more to “his thesis [being] contrary to established, albeit unsubstantiated, dogma,” they also observed:
Unfortunately, when he was compelled to identify these elusive organisms, particularly when dealing with regulatory agencies, he meandered around known pathogens, unable to build a strong case for any of them. His main argument, and the strongest one he could marshal in all his writings in favor of the infectious etiology of these clinical entities, was the response to treatment that he had witnessed.
A good argument can be made that characterizing the “bug” not only is, but ought to be a sine qua non for treating a putative infectious disease with drugs. This is true because the drugs are not benign or universally effective, as argued above, but also because there are precedents suggesting that to do otherwise opens the door to mistreatments. Bacteria or viruses are frequently offered as potential etiologic agents for all sorts of diseases whose causes are poorly understood, particularly when there is an inflammatory component (that is one reason that the H. pylori hypothesis didn’t represent a new “paradigm”). Osteoarthritis and rheumatoid arthritis are examples, but these have, so far, eluded attempts at proof.
When I first learned about sarcoidosis and Crohn’s disease in medical school in the 1970s, I would have bet dollars-to-donuts that they, and rheumatoid arthritis and a few other diseases for that matter, would eventually be shown to have infectious etiologies. I would still almost make that bet, my only hesitation being that after 30+ more years of investigations and impressive advances in microbiology (including vastly more powerful methods of detecting well-veiled foreign invaders, from electron microscopy to nucleic acid amplification), no apparent culprits have feen found.
During that same time, moreover, not only peptic ulcer disease but also Lyme disease, Legionnaire’s disease, and Toxic Shock Syndrome were shown to have bacterial origins, and AIDS was shown to be caused by a virus. Thus on the basis of what has been learned about infectious diseases it can’t be argued, with a straight face, that biomedical progress is hampered by stodginess or petty jealousies or dogmatic thinking or conflicts of interest or any of the other usual suspects, even though they certainly all exist among individual scientists. What hampers progress, in cases such as Likoudis’s, is what hampers all scientific progress: the context is not prepared.
Instances in which at least some people have become convinced of a spurious infectious etiology, on the other hand, have not been pretty. In the early 20th century, prior to the discovery of antibiotics, some psychiatrists became convinced that “insanity” was caused by bacteria in the mouth and that the appropriate treatment was “surgical bacteriology”: tooth extractions, tonsillectomies, and in intractable cases removal of “testicles, ovaries, gall bladders, stomachs, spleens, cervixes, and especially colons.” More recently a putative bacterial cause of atherosclerosis has spawned a small quack industry.
Even if any of the diseases mentioned above is eventually found to be infectious in origin, this will not necessarily vindicate those whose premature exuberance put patients in harm’s way. Such exuberance ought to motivate legitimate investigations, not half-assed, ill-conceived treatments. Still, I seem to hear a ghostly voice in my ear, speaking Italian with a thick Greek accent…
E pur si muove!

   The case of John Lykoudis and peptic ulcer disease revisited: Crank or visionary?
David Gorski
Link
One of the themes of SBM has been, since the very beginning, how the paradigm of evidence-based medicine discounts plausibility (or, perhaps more appropriately, implausibility) when evaluating whether or not a given therapy works. One of our favorite examples is homeopathy, a therapy that is so implausible on a strictly scientific basis that, for it to work, huge swaths of well-established science supported by equally huge amounts of experimental and observational evidence would have to be found to be all in serious error. While such an occurrence is not per se impossible, it is incredibly unlikely. Moreover, for scientists actually to start to doubt our understanding of chemistry, biochemistry, pharmacology, and physics to the point of thinking that our understanding of them is in such serious error that homeopathy is a valid description of reality, it would take a lot more than a bunch of low-quality or equivocal studies that show no effect due to homeopathy detectably greater than placebo.
On Friday, Kim Atwood undertook an excellent discussion of this very issue. What really caught my attention, though, was how he educated me about a bit of medical history of which I had been completely unaware. Specifically, Kim discussed the strange case of John Lykoudis, a physician in Greece who may have discovered the etiology of peptic ulcer disease (PUD) due to H. pylori more than a quarter century before Barry Marshall and Robin Warren discovered the bacterial etiology of PUD in 1984. One reason that this story intrigued me is the same reason that it intrigued Kimball. Lykoudis’ story very much resembles that of many quacks, in particular Nicholas Gonzalez, in that he claimed results far better than what medicine could produce at the time, fought relentlessly to try to prove his ideas to the medical authorities in Greece at the time, and ultimately failed to do so. Despite his failure, however, he had a very large and loyal following of patients who fervently believed in his methods. The twist on a familiar story, however, is that Lykoudis may very well have been right and have discovered a real, effective treatment long before his time.
Kimball is right to point out that discoveries require context. Medicine can be prone to dogma. Of that, there is no doubt. Sometimes, physicians resist challenges to prevailing medical dogma. Of that, too, there is no doubt. However, reading the story of John Lykoudis, I couldn’t help but wonder the full context of his activities and efforts to convince the Greek medical authorities of the 1950s and 1960s that he was on to something. It also allows me to indulge myself in a bit of my surgical training, the bulk of which, ironically enough, occurred during the time period when the discoveries of Warren and Marshall were first revolutionizing the treatment of PUD and becoming increasingly accepted in the 1990s. When evaluating a story like that of Dr. Lykoudis and why he was unable to convince the medical profession of 50 years ago that his findings had merit, it’s very important to understand (1) the prevailing notion at the time of PUD etiology and, even more importantly, (2) how PUD was diagnosed and treated in the 1950s and 1960s.
Throughout most of the 20th century, PUD was thought to be caused by stress and dietary factors. The primary treatment for disease that had not yet developed complications was bed rest and prescription of bland diets. Towards the mid-20th century it became increasingly appreciated that gastric acid is a major factor in the etiology of PUD, and increasingly antacids, then later H2 receptor antagonists like cimetidine and ranitidine, and then still later proton pump inhibitors like omeprazole. Cimetidine and ranitidine were the mainstays of PUD treatment when I was a medical student and a resident. For complicated PUD that either was not adequately managed by drugs, sometimes surgery would be necessary. Endless were the discussions and arguments among surgeons fo the time what operation was best for PUD management, a Bilroth I or Bilroth II, vagotomy and pyloroplasty, and, at the time I was finishing my residency, the highly selective vagotomy. Then, of course, there were the discussions of what to do in the case of a bleeding ulcer that couldn’t be stopped using endoscopy or what operation to perform in the case of a perforated ulcer.
One must also remember that in the 1950s and 1960s, the diagnosis of PUD was much different than it was even in Marshall and Warren’t time in the 1980s. That’s because in the 1970s and 1980s fiberoptic endoscopy became the dominant method of diagnosing PUD. Not only did it allow for direct visualization of the ulcer, but it also allowed for potential therapy without laparotomy. More relevant for purposes of this discussion, endoscopy allows isolation of the H. pylori that causes PUD. Prior to that, the methods of diagnosis were not as accurate. In general, PUD would be diagnosed by history, physical, and then an upper GI X-ray series, in which the patient would swallow barium-containing slurries, after which fluoroscopy would visualize the stomach and duodenum.
Consequently, if you imagine yourself back in the 1950s and 1960s, the time when Dr. Lykoudis practiced, the diagnosis of PUD was less certain than it was in the 1980s. Unless it was severe, it might not even show up on upper GI series, given the limitations of the imaging technology of the time. It wasn’t always possible to distinguish between gastritis and true PUD. Consequently, a lot of diagnosis and treatment of PUD in pre-endoscopy times ended up being a lot more empiric than it is now. The diagnosis was arrived at clinically, and it was symptomatic relief that was used as the main measure of treatment success.
So what would it have taken to demonstrate, given the technology of the time, that Dr. Lykoudis’ treatment was effective? It would have taken pretty much what it would take today, minus the use of endoscopy. Consequently, we could envision a clinical trial in which standard of care at the time (bland diet, antacids, etc.) was tested against the standard of care plus Dr. Lykoudis’ antibiotic regimen. The design would be simple: Two groups, one receiving standard of care plus placebo, one receiving standard of care plus antibiotics. Alternatively, although it probably wouldn’t be considered ethically acceptable now, back in the 1960s, it probably would have been considered ethical to do an alternative study, directly comparing the then standard of care to Dr. Lykoudis’ antibiotic regimen. Whichever design were chosen, before entry in the study, each patient would have to undergo upper GI to document the presence of PUD and measure its severity. They would also have to undergo upper GI at the end of the study to document healing. Throughout the study, pain scores would be measured, and patient outcomes tracked to see what proportion of patients end up requiring surgery for their PUD and which proportion can go back to eating a regular diet. Unfortunately, Dr. Lykoudis didn’t have data anywhere near that level of rigor.
On the other hand, what I’m describing above is the equivalent of what we would call today a phase III clinical trial. In general, phase III trials aren’t started without preliminary data. But what sort of preliminary data are usually needed to provide adequate scientific justification for a phase III trial? Today, we require preclinical evidence in the form of cell culture and animal studies, as well as lesser levels of clinical evidence, such as smaller trials like phase I or phase II trials. The preclinical data provide scientific justification and plausibility, and the human data bolster that. Of course, we can’t apply today’s standards to the science of 50 years ago. Back then, the methodology and ethics of randomized clinical trials were not as advanced and well-worked out as they are today.
So what happens if we look at Dr. Lykoudis’s story and whether he might have had enough data to justify a large phase III-like clinical trial in the time in which he was practicing? By today’s standards, he probably did not. As Kimball pointed out, although plausibility does not mean understanding the mechanism, it does mean that there should be a potentially plausible mechanism, as there does not exist, for example, for homeopathy. The concept that there might be a bacterial cause of PUD is not, on the surface, incredibly implausible. But was it plausible enough to justify a clinical trial of the sort that Dr. Lykoudis proposed? What would it take? This is what Dr. Lykoudis proposed:
In 1967, Lykoudis succeeded in getting the attention of the Prime Minister’s office. His correspondence with the Minister of Health on 21 August, 1967, a sad document indeed, is revealing. He registers his frustration that medications with apparently no effect on PUD were approved, whereas Elgaco was repeatedly rejected. He proposes, in essence, a phase III trial: 100 PUD patients to be treated at a State hospital by the eminent professors, 50 with conventional treatment and 50 with Elgaco. ‘Their refusal to approve it is understandable, but their refusal to test it is not!’ he writes.
A not unreasonable assertion. Unfortunately, as Kimball pointed out, the technology to isolate and culture H. pylori didn’t exist at the time. Absent that, providing strong evidence for a bacterial etiology for PUD would require an obvious and strong response to antibiotics in the form of unambiguous symptom relief and healing documented on followup upper GI imaging studies. Even if Lykoudis had had that, he would have been unable to culture the organism responsible for PUD, which would have left scientists in a quandary. Response to antibiotics is storngly suggestive of an infectious etiology, but, absent an organism, one can never determine for sure whether it is in fact a bacteria causing a disease or the antibiotic has an activity other than its ability to kill bugs, an additional activity. For example, erythromycin increases GI motility. Thus, in the context of the time, it’s not surprising that Lykoudis’ ideas were considered highly implausible, and it would have required very strong evidence to make the idea seem plausible.
Steve Novella made a cogent observation:
But taken at face value, I think the real lesson is that process is more important than whether or not one turn’s out to be correct. Science is about process.
The problem with Lykoudis is that his behavior was indistinguishable from the myriad quacks and charlatans that existed then and exist today. That in hind sight one turned out to be on the right track is not all that surprising, and their contemporaries should not be faulted for their inability to predict the future.
The question is – what did Lykoudis do to convince the scientific community of his claims. Did he perform carefully controlled double-blind placebo-controlled trials? Did he attempt to enlist the help of a microbiologist to try to isolate the organism? Or did he just expect people to take his word for it?
What did he do to deserve being taken seriously? Being right in the hindsight of history is not enough.
And I think that this is the key point. Science is a process. It is by its very nature constrained by what is known at any given time in history. In the context of Dr. Lykoudis’s time and given what was known then, it is not surprising that his idea would have encountered heavy resistance from the scientific orthodoxy of the time. Another issue to consider is regional variation in physicians’ attitudes. One example I like to use to illustrate this is the reaction of European physicians to the ideas of Ignaz Semmelweis. Semmelweis, as most readers will recall, first demonstrated that the high rate of puerperal fever in the obstetrics ward run by physicians was due to physicians not washing their hands after doing autopsies, going straight from the morgue to the delivery room. Semmelweis’ findings were far more favorably received in England, for example, than they were on the continent. It is possible, although by no means assured, that Lykoudis’ ideas might have been better received if he had lived in a different part of the world.
Here’s one final consideration. Science is performed by human beings. Although it is a process designed to overcome human biases, communicating the results of scientific research is subject to the same idiosyncracies to which any human communication is subject. Anyone who’s ever been to a scientific conference knows that. It is quite possible to be right and, to put it bluntly, to piss off the very scientists that need to be convinced so much that they harden their positions protecting the scientific consensus. One example is Semmelweis himself. Sherwin Nuland, in his book The Doctors’ Plague: Germs, Childbed Fever and the Strange Story of Ignac Semmelweis, suggested that if Semmelweis had communicated his findings more effectively and managed not to antagonize the medical establishment so thoroughly he might not have been marginalized and dismissed in the manner that he was. At the very least, he may not have been met with as much hostility. Galileo, although apparently not intentionally, alienated the Pope at the time by putting his words in the mouth of a character named Simplicio defending the Aristotelian Geocentric view in Dialogue Concerning the Two Chief World Systems. Reading between the lines, one can see echoes of this sort of antagonism in Lykoudis’s story. Indeed, Michael Phillips of St. Vincent’s Medical Center wrote in an letter to The Lancet:
I propose a less dramatic metaphor: medicine is a marketplace of ideas, with sellers and buyers. Sellers (innovators with new ideas) advertise their intellectual property to potential buyers (other physicians). This buying public is highly sophisticated and sceptical. Quite correctly, physicians will only accept the highest quality new ideas because the lives of their patients are at risk. So physicians buy only when they see the hallmark of quality: publication in a respected peer-reviewed journal.
That in a nutshell explains the tragedy of John Lykoudis. He had a wonderful intellectual product: the insight that peptic ulcer disease is infectious, supported by the evidence that it can be cured with antibiotics. But he lacked the ability (or the training) to sell this insight to his colleagues. He was a retail trader who treated individual patients instead of targeting the wholesale market of other physicians.
Of course, this is the problem. Most physicians are not scientists, and many physicians are very prone to being swayed by anecdotal evidence. Without well-designed clinical trials based on the best basic and translational science available, way to validate or refute anecdotal data. I find Lykoudis’s story to be a cautionary tale. Whether he was correct and thus the true “Galileo” of H. pylori, rather than Warren and Marshall or whether he was just another crank, his story demonstrates that we scientists should be very careful to guard against excessive smugness. As has been repeated by many skeptics in many variants over the years, it is not sufficient to claim the mantle of Galileo as a persecuted martyr for science. You must also be right. Even though it is not clear whether, taken in the context of the time, Lykoudis was a crank or a misunderstood physician who was ahead of his time, Warren and Marshall’s vindication of his ideas that PUD is bacterial in etiology reminds us that not all who claim the mantle of Galileo are necessarily cranks. The vast majority usually are, but on very rare occasions we do see a real Galileo.

   Antidepressants and Effect Size
Harriet Hall
Link
Antidepressant drugs have been getting a bad rap in the media. I’ll just give 3 examples:
    • On the Today show, prominent medical expert Tom Cruise told us Brooke Shields shouldn’t have taken these drugs for her postpartum depression.
    • In Natural News, “Health Ranger” Mike Adams accused pharmaceutical companies and the FDA of covering up negative information about antidepressants, saying it would be considered criminal activity in any other industry.
    • And an article in Newsweek said “Studies suggest that the popular drugs are no more effective than a placebo. In fact, they may be worse.”
Yet psychiatrists are convinced that antidepressants work and are still routinely prescribing them for their patients. Is it all a Big Pharma plot? Who ya gonna believe? Inquiring minds want to know:
    • Are antidepressants more effective than placebo?
    • Has the efficacy of antidepressants been exaggerated?
    • Is psychotherapy a better treatment choice?
The science-based answers to the first two questions are clearly “Yes.” The best answer to the third question is “It depends.”
In 2008, Erick Turner and four colleagues published an article in The New England Journal of Medicine (NEJM) entitled “Selective Publication of Antidepressant Trials and Its Influence on Apparent Efficacy.” The FDA is able to make sure that drug companies don’t pick and choose which trials, and which outcomes within those trials, get seen. Using clinical trial data from the FDA as a gold standard, Turner, et. al. examined how these same trials were reported in published journal articles, They found that:
…according to the published literature, the results of nearly all of the trials of antidepressants were positive. In contrast, FDA analysis of the trial data showed that roughly half of the trials had positive results.
And some of the negative trials were published with a “spin” that made them appear positive. The data did show that each drug was superior to placebo, but the true magnitude of that superiority was less than a diligent literature review would indicate. They warned that
By altering the apparent risk–benefit ratio of drugs, selective publication can lead doctors to make inappropriate prescribing decisions that may not be in the best interest of their patients and, thus, the public health.
Irving Kirsch has been outspoken about antidepressants’ alleged lack of efficacy. In a controversial meta-analysis published in 1998, he found that placebos provided approximately 75% of the improvement provided by active drug. He suggested that the other 25% is debatable and could be due to an enhanced placebo response when patients experience side effects that convince them they are getting an active drug. In a further study in 2002, he “questioned the clinical significance of antidepressants.”
Kirsch recently looked at the FDA data for 4 of the 12 drugs that Turner examined. In spite of the smaller sample, where Turner found an effect size of 0.31, Kirsch got 0.32. So they got almost exactly the same result. But it was their interpretations of that result that were very different. Kirsch concluded that antidepressants are ineffective, while Turner found that the drugs were indeed superior to placebo. As the figure below shows, each drug’s effect size was positive. Also, none of the confidence intervals overlapped zero. This means that, while there is some probability that the true effect size is zero, meaning that antidepressants and placebo are equal in efficacy, that probability is negligibly small.
The discrepancy between Turner’s and Kirsch’s interpretations hinges on what these effect size numbers mean in terms of clinical significance,. Values of 0.2, 0.5, and 0.8 were once proposed as small, medium, and large effect sizes, respectively. The psychologist who proposed these landmarks admitted that he had picked them arbitrarily and that they had “no more reliable a basis than my own intuition.” Later, without providing any justification, the UK’s National Institute for Health and Clinical Excellence (NICE) decided to turn the 0.5 landmark (why not the 0.2 or the 0.8 value?) into a one-size-fits-all cut-off for clinical significance. In an editorial published in the British Medical Journal (BMJ), Turner explains with an elegant metaphor: journal articles had sold us a glass of juice advertised to contain 0.41 liters (0.41 being the effect size Turner, et al. derived from the journal articles); but the truth was that the “glass” of efficacy contained only 0.31 liters. Because these amounts were lower than the (arbitrary) 0.5 liter cut-off, NICE standards (and Kirsch) consider the glass to be empty. Turner correctly concludes that the glass is far from full, but it is also far from empty. He also points out that patients’ responses are not all-or-none and that partial responses can be meaningful.
Incidentally, NICE is no longer using the 0.5 effect size cutoff.
If we followed Kirsch’s interpretation and rejected antidepressants, how would we treat depression? Psychotherapy avoids the side effects of drugs, but it has its own drawbacks: it is expensive, time-consuming, and variable in quality. How effective is psychotherapy? Psychotherapy trials also suffer from publication bias, just like antidepressant drugs. And when one weeds out low quality studies, psychotherapy has an effect size of only 0.22, lower than the value for antidepressants reported by Kirsch himself, So if we reject any treatment below the (arbitrary) 0.5 cutoff, when a mental health care provider is faced with a patient in need of help, is he or she to do nothing at all?
I don’t doubt that antidepressants have sometimes been over-prescribed and used inappropriately for lesser levels of depression where they are less effective or even ineffective, but this is probably true for psychotherapy, as well. On the other hand, it has been estimated that only about half of depressed patients are getting any kind of treatment. Severe depression is a life-threatening disease. A recent study showed that antidepressants reduced the risk of suicide by 20% in the long term. The risk/benefit ratios are still not clear cut for either form of treatment.
Once more, science fails to give us the black-and-white answers we crave. And once again we are reminded that we can’t rely on the media for accurate, nuanced information about medical science.
For his assistance in preparing this article and for providing the figure, I want to thank Erick Turner, M.D., Department of Psychiatry, Oregon Health and Science University; Staff Psychiatrist, Portland Veterans Affairs Medical Center; Former reviewer, FDA.

   Oh yeah? Thalidomide! Where’s your science now?
Scott Gavura 
Link
Online discussions on the merits of alternative medicine can get quite heated. And its proponents, given enough time, will inevitably cite the same drug as “evidence” of the failings of science. Call it Gavura’s Law, with apologies to Mike Godwin:
As an online discussion on the effectiveness of alternative medicine grows longer, the probability that thalidomide will be cited approaches one.
A recent comment on my own blog, regarding the homeopathic product Traumeel, is typical:
If the scientific method is all that separates an accepted claim, ie Thalidomide, Vioxx, Bextra, Darvon, from mere anecdote, of what benefit is the Science?
As a non-scientist consumer, I’ll take the anecdotes and my own experience. Thank you.
If scientists want to be taken seriously, they must stop selling themselves to the highest bidder becoming corporate whores without a shred of decency. To my mind, that’s how the claims for Thalidomide, Vioxx, Bextra, Darvon were accepted, making the scientific method utterly worthless.
To this commenter, “science has been wrong before.” And that invalidates science, and apparently validates homeopathy. It’s a fallacious argument. But does thalidomide actually represent a failing of science-based medicine? No, not even close. It’s so wrong, it’s not even wrong. Thalidomide is good example of the importance of science-based medicine and why allowing alternative medicine to be sold in the absence of good science is a concern.
The broad strokes of thalidomide causing thousands of birth defects are well known. But the details are important to understand the implications to regulation, and to science-based medicine, today.
Thalidomide was first marketed in 1958 in Germany by Chemie Grünenthal. While its developmental origins are somewhat unclear, it was initially marketed as a treatment for seizures, and later as an anti-nauseant and sedative. At the time, barbiturates were frequently used as sedatives. And compared to barbiturates, which were highly toxic in overdose, thalidomide was well tolerated, even in overdose. Based on its apparent safety, no prescription was required. Eventually its attractiveness as an anti-nauseant led to its use in pregnancy for morning sickness.
It wasn’t known at the time, but fetal exposure to thalidomide between days 35 and 48 was causing severe limb and organ defects in 20-30% of children. In the 1950′s it wasn’t even recognized that drugs could cross the placenta and cause adverse effects to the fetus. Thalidomide hadn’t been tested on pregnant animals prior to marketing for use in pregnancy. Regulators didn’t require it. And the consequences were horrific.
Thalidomide became a popular drug because of its apparent safety and effectiveness, and it was marketed alone and in combination with other drugs in Germany, the UK, Canada, and other countries. But within a few years, babies started being born with characteristic limb and organ deformities. In 1961, two independent researchers identified thalidomide as the likely causal agent. The manufacturer sought to undermine and discredit the findings, but it was clear – the drug had caused catastrophic harm to thousands of fetuses. Subsequent animal testing confirmed this.
The USA largely escaped the thalidomide tragedy – due to stronger regulations, and the action of a single employee.
American drug regulations have evolved over time. The Pure Food and Drugs Act (1906) did not require that drugs be tested for safety and efficacy before sale. It was the regulator’s responsibility to demonstrate a product was unsafe. In 1938, the S.E. Massengill company manufactured a liquid version antibiotic sulfanilamide, mixing it with diethylene glycol -a poison more commonly used as antifreeze. When patients started dying, a massive recall was implemented – but not before over 100 people died. The tragedy brought about significant reforms to drug regulations in the form of the Food, Drugs and Cosmetics Act (1938) that required manufacturers to obtain FDA approval for any drug sold, prior to any sale.
Frances Oldham Kelsey was a reviewer with the FDA in 1960, and was responsible for evaluating the thalidomide marketing application. Kelsey refused to approve the drug in the absence of safety data. Her concern was peripheral neuropathy – not teratogenicity, which wasn’t even considered at the time. She described the application process in a paper she published in 1965:
The New Drug Application for thalidomide was presented in September, 1960. The drug had been marketed in Germany since 1957 where it was available without prescription, and in Great Britain since 1958. However, it was felt that the evidence submitted in the application was not adequate to indicate the safety of the drug. In particular, although this drug appeared to be remarkably nontoxic in animals and human beings, little or no information was available concerning its absorption, distribution in the body, or its excretion. Since the possibility existed that the low toxicity of the drug in certain species might be related to poor absorption in those species, and that under certain conditions the absorption in other species might be increased, further work was requested relative to the metabolism of the drug.
As the FDA studied the side effect profile, the teratogenicity of the drug became clear as and the application for licensure was discontinued. Kelsey was hailed as hero. Yet despite the refusal to formally approve the product for sale, millions of doses had been administered as part of “clinical trials” (in name only). Yet due to Kelsey’s refusal to approve the drug, the drug was not widely used, and only 17 children were born in the USA with thaldomide-induced effects.
Regulators worldwide acted to ensure another thalidomide tragedy would not occur. In 1962 the Kefauver-Harris Drug Amendments were passed, increasing the rigor of the drug approval process, requiring the demonstration of safety and effectiveness via objectively designed and executed clinical trials. These amendments also implemented the requirement for adequate preclinical trials before any human studies, including animal studies to evaluate fetal risks. Marketing requirements were also implemented, restricting manufacturers from making unfounded, unsubstantiated claims of safety and efficacy. Finally, the amendments ushered in the requirement for manufacturers to collect and report all adverse events associated with drug use – measures designed to capture any events not otherwise identified in pre-marketing clinical trials. Similar actions occurred in other countries. Today, over 100 countries now collaborate with the World Health Organization to pool adverse event reporting and look for signals of drug harms.
Lessons Learned
The worldwide disaster of thalidomide led to most of the current framework we have in place today to evaluate the safety and efficacy of drug products. Ironically, these are the same requirements that manufacturers of supplements and other “alternative” medicine products have been largely successful in circumventing. In the USA, it’s DSHEA, which (as blogged about regularly at SBM) removed the onus of demonstrating safety and efficacy from the manufacturer and put the requirement to demonstrate harm on the FDA – exactly the same scenario as drugs in the early 1900′s. In Canada, the Natural Health Product regulations has introduced a lowered bar for non-drug supplements: Today even homeopathic remedies are deemed safe and effective and approved with unique recommended uses.
The Resurrection of Thalidomide
Almost 30 years after thalidomide was withdrawn, it’s back on the market in the United States, Canada and many other countries. It’s been found to be effective for a number of conditions including erythema nodosum leprosum, multiple myeloma, and is being investigated for efficacy in an array of other conditions. There’s an analog of thalidomide now marketed, lenalidomide, also used to treat cancer. To minimize the teratogenicity risk, intensive programs are in place to minimize any possibility of use in pregnancy. So while the drug has known harms, it seems to be highly effective for some medical conditions where few effective alternatives exist. Access, therefore, is based on a careful evaluation of the risks and benefits.
Does Thalidomide Invalidate Science-Based Medicine?
Cases like thalidomide provide a good example of why SBM authors argue against regulatory double-standards, and advocate for a single, science-based standard for evaluating all products: drugs as well as supplements. There is no intrinsic reason to think any product, regardless of its source, is safe or effective – we must evaluate it, objectively. Thalidomide’s history is a cautionary tale reminding us that assumptions without good evidence can lead to terrible consequences.
And that leads to the logical fallacies that the thalidomide retort represents. Thalidomide is an effective drug for some conditions – but it is a teratogen. It’s not alone in this regard – other drugs have been identified as teratogens. The sale of any drug is based on an evaluation of the known benefits and risks.And getting back to the comment I received at the top, those that support alternative medicine will reject the science-based approach: This is the system that gave us thalidomide (and Vioxx, etc). Therefore, it’s a failure, and we should reject it. This is the perfect solution fallacy.
Admittedly, medicine is not perfect. Regulators don’t identify all the harms before a drug is licensed. Drug manufacturers can behave badly. And drugs don’t always work the way we want them to, and they can cause harms. They commenter I cite above took objection to an evaluation of homeopathy. Yes, homeopathy has no toxic side effects (usually) and causes no teratogenicity – but it also has no demonstrated efficacy beyond placebo effects. The harms and problems of science-based approaches add no support to the efficacy claims of any alternative medicine system. Homeopathy, acupuncture, and reiki don’t become effective because drugs can have side effects. SBM may not be perfect, but it delivers the goods.
More generally, comparing science-based medicine to any alternative medical system a is false dichotomy. Unlike the different alternative medicine systems (homeopathy, naturopathy, reiki, etc) which are typically based around a fixed set of rules, the only thing that science-based medicine interventions have is common is that they work. SBM is not immutable to change – treatment shown to be safe and effective become medicine, and those that are not, are discarded. So although thalidomide causes birth defects, it is also an effective medication when used properly. We don’t have to make a choice between a system that produced drugs that cause birth defects, and alternative medicine. We choose to use treatments where the expected benefits outweigh the known risks.
Citing thalidomide as an argument against science-based approaches is also a straw man argument. The way drugs are regulated today bears little resemblance today to when thalidomide was licensed and sold. Safety standards are far more rigorous, in part because of the lesson of thalidomide. Today, the only products that are not subject to these same strict safety and efficacy standards are usually the alternative medicine treatments.
Citing thalidomide is also an appeal to fear. We hear the word and we immediately think of children born with birth defects. We don’t want that to happen. But like the perfect solution fallacy, the catastrophic side effects of thalidomide in pregnancy add no merit to to claims of efficacy of any alternative medicine treatment or system. We still need the evidence.
Conclusion
Thalidomide was a very real tragedy with a huge human cost. It’s most important lesson is that assumptions of safety and efficacy, in the absence of evidence, can be catastrophic. Citing thalidomide as a a reason to reject science-based medicine is not only fallacious, it reflects a fundamental lack of understanding of the lessons learned.

   Deadly Indeed
Mark Crislip 
Link
There are sources of information I inclined to accept with minimal questioning. I do not have time to examine everything in excruciating detail, and like most people, use intellectual short cuts to get through the day. If it comes from Clinical Infectious Diseases or the NEJM, I am inclined to accept the conclusions without a great deal of analysis, especially for non-infectious disease articles. Infectious disease publications I have to read more closely; its part of passing as an expert.
Outside of medicine, I am predisposed to accepting at face value many of the articles in Skeptic and Skeptical Inquirer. They are trusted sources. Some topics, like haunted house or Big Foot investigations, I barely skim. After all these years, I doubt there will be any new insights into the subject. Other topics, depending on my interest, I may read more carefully.
I often read longer articles many times. First a quick skim to see if it offers anything of interest. If it does, then I may read it carefully.
This months Skeptical Inquirer had an article called Seven Deadly Medical Hypotheses by Reynold Spector. Just seeing the title and knowing the magazine, I was primed to accept the content at face value. I enjoy a well reasoned, thoughtful rant. I relish a clever diatribe, even if I do not agree with the topic. So I gave it a quick skim. I was discomfited. My first gut check was ick. But I was uncertain why. So I read it slowly and carefully, and still ick. But why?There is a degree of self absorption in being a blogger. I can write about what I want any way I want (I remain amazed at how much I can get away with). The process of writing about a topic helps me clarify in my own mind issues with articles.
The author of 7, as I shall refer to the article, has over 200 published articles, is a former executive vice president in charge of drug development at Merck and oversaw the development of 15 drugs and vaccines. I am nobody from nowhere who just takes care of infected patients for a living. He wins the argument from authority; I am the E. coli evaluating the human. Oh well, this is more an exercise for me to enlighten myself; you are the innocent bystander.
Overall the tone of 7 ? It reminded me of the Health Ranger at NaturalNews.com. Really. Lots of dramatic statements, no qualifiers, no buts, no subtlety, no nuance. To me, what marks good medical writing is an understanding that there is far more grey than black and white and that generally people are doing the best they can within numerous limitations One of the many characteristics of the Health Ranger is hyperbole without nuance. The Health Ranger has a belief system and sees the medical industrial complex through that lens; information is used to support a predetermined conclusion. Health Ranger is a bombastic style that is both self assured and self referential.
Let us see what 7 has to say. It begins
A chronic scandal plagues the medical and nutritional literature: much of what is published is erroneous, pseudoscientific, or worse.
I’ll grant the first. I am an Ioannidis convert. The second seems hyperbole and exaggeration. Pseudoscientific? Like homeopathy, psi and astrology? Sorry The author is 17 words in and he has lost me. I already question his veracity and judgement. I read the literature. Hundreds of papers a month. I know the literature, and Sir, it is not pseudoscientific. Suboptimal, often, but not pseudoscientific. The third? What could be worse than pseudoscientific? Oh yeah. Wakefields Lancet article. But fraud is a very rare exception in the over 20 million references on Pubmed. The author’s opening salvo strikes me as someone more interested in polemic than truth. If done with verve and panache, and above all wit, I like a good polemic. Pomposity with hyperbole, not so much, and calling the medical literature erroneous, pseudoscientific, or worse leans towards the latter.
Two major factors account for a large proportion of this problem. First, many medical and nutritional hypothesis are ill-conceived.
Are they? Over 20 million references in Pubmed. A few, perhaps, were ill conceived before they are tested. Say, measles vaccine induced gastroenteritis causing autism? Not even that. If approached honestly and competently, it would be a long shot, but you never knows unless you look. That is what a great deal of medical research is about: looking around to see if an etiology or intervention or medication will be effective. Most ideas, I would guess, go nowhere.
Second, the methods used are often epistemologically unsound.
Got me there. What is epistemologically unsound? Even after looking epistemologically up on the interwebs, I am uncertain what it means. I expect the comments will school me on the meaning of epistemologically unsound. I guess that is why I am a lowly clinician.
Moreover, the same unsound methods are often repeated multiple times on the same tired hypotheses with the same incorrect results.
Isn’t that three major factors? Or is that the unsound epistemological I cannot understand? I shouldn’t quibble about counting, but I feel a rising tide of ridicule and scorn, and I am not one to hold it back.
I am not even done with the first paragraph, and the author has epistemologically lost me. Maybe there is good reason to be unsettled with the article. And in the first five sentences, there are four references, all to works by the author, to justify the position. I tend to prefer external references in my literature; the hyperbolic self-validation is what I expect from the Health Ranger and his ilk. But again, who am I to question (1)?
… there is an epidemic of published studies that do not follow the principles of sound medical science- the principles demanded by the US Food and Drug Administration for the licensure and sale of medications.
Well, most studies are preliminary and exploratory. The rigor demanded by the FDA is the final step in a long process starting with basic principles and, perhaps, epidemiology. I can’t imagine we should jump to huge randomized, placebo controlled trials for every therapy and to answer every question. Seems a wee bit excessive to me. Start small and build. The downside is that there will be dead ends and false conclusions. The upside is that in the end, a close approximation of Truth will be determined.
The resulting “findings” of such misleading or erroneous studies are often hyped by the news media on the day they are reported or published without any additional, careful analysis.
Hyped “findings”? Nothing like that in the first two paragraphs of the this essay. Nope. Nothing to see there but a well reasoned, careful, nuanced prologue for the body of the essay. “I” am always “mistrustful” of people who use “quotes” as a form of “sarcasm” when sarcasm is not used for good “effect” like “humor” because it otherwise comes across as “supercilious”.
Now I am starting to understand my discomfiture. Still, that’s just the first two paragraphs. The body will better, right?
The author then proceeds to the background of how to do a good study: generate a plausible, testable hypothesis and test it. He uses the Scandinavian Simvastatin Survival Study as an example of medicine done right (a Merck product if you care) and bemoans that not every study meets this high standard.
Too many published studies fail to adhere to these high scientific standards and lead to faulty, and even dangerous, conclusions.
Which is true and to my mind understandable, since there are not the resources to do perfect studies of every hypothesis. Not every car is a Lexus, not every restaurant has a Michelin three star rating. You can’t always get what you want (2). The issue to my mind is not that there are suboptimal studies; they are often used to find search for hypothesis that can be tested in better trials. A large part of research is flailing about looking for something interesting to investigate in further detail. Not everyone has the resources to test everything using the ‘hypothetical/deductive method” to answer all out questions, like the FDA demands. Although this is not always the preferred method of generating ideas to test. I don’t need quotes to cast aspersions on the validity of information or generate guilt by association. I have learned a thing or two from reading the Health Ranger.
I wonder how many suboptimal studies it required to get to the point of the Scandinavian Simvastatin Survival Study? The concepts to be tested did not appear from the void, fully formed. The author does not, as will be seen, pay attention to the history and context of the evolution of medical ideas.
The author then proceeds to his 7 deadly hypotheses. Well, one deadly, 6 not so much. But guilt by association is a game played by the author of 7 as well.
1) the investigator does not need a specific hypothesis and/or can use an inadequate method to test hypothesis.
He uses the example of epidemiology generated by case-control and cohort studies (the kind of studies that lead to the simvastatin study) and the effects of hormone replacement therapy. He points out that these epidemiologic studies, for a variety of reasons, can lead to erroneous conclusions. Fine. The other option? With no preliminary studies jump straight to a huge trial? And sometimes epidemiology can lead to important results: that a certain water pump is the epicenter of cholera or that chimney sweeps have more testicular cancer. Or that lowering cholesterol is associated with a decrease in vascular deaths.
Epidemiology is part of a continuum of understanding and evolution of medical knowledge. But strawmen are easier to burn than recognizing the stuttering, somewhat chaotic progress of medical knowledge. If proving a point is more important than understanding complexity, this is how you argue.
He then proceeds to genome-wide association studies (GWAS) that have been a disappointment for elucidating genetic causes of heart disease and Alzheimer’s. The author considers GWAS a failure. I suppose if you have a narrow perspective, yes it has been a failure. So far. Huge amounts of information about the genome have been generated, and I am always a fan of knowledge for knowledge sake. In the world of infectious diseases, there are single gene polymorphisms in the immune system that can increase or decrease a patients risk for a variety of infections. Is it of clinical relevance yet? No. Is it interesting? Oh, yes. Will it lead to a new treatments and diagnostic interventions in the future? Who knows. But trying new ideas may fail but still lead to insights that may lead to better interventions. I would wonder what secondary advances in technology and understanding were accomplished as a results of the GWAS studies.
It is like complaining that the Apollo program only put 12 people on the moon so the program is a bust since we are not going to the moon for vacation. Here is a dirty little secret from a mere clinician. I learn far more from failure than I ever have from success. “The most exciting phrase to hear in science, the one that heralds new discoveries, is not ‘Eureka!’ (I’ve found it!), but ‘That’s funny…’ -Isaac Asimov.” If you are a clinician, it is not ‘That’s funny,’ but ‘Oh shit’ that really drives change and knowledge.
2) If women replace these missing hormones post menopausally with HRT, they will remain “youthful” and not suffer from heart disease, dementia, vaginal dryness, hot flashes, and fractured bones.
I remember the late ’80′s, a time that was the heyday of HRT, when I was in my internal residency training and discussing the issues at length not only in clinic, but with my mother. I remember discussing the epidemiologic data and the worries of cancer. The author states that
…based on these (biased) studies, false claims were made the HRT protected against cardiovascular disease and dementia.
As if we knew it was false at the time. It was the best guess based on the data, and epidemiology can give insights that can be later confirmed by better studies. He also says
“the proponents…ignored the well-documented fact that estrogen is a carcinogen that causes breast cancer that can kill women” and that “HRT caused a 25% increase in breast cancer.”
I do not know where the author was practicing, but I remember talking with patients (I know, flawed memory) and my mother about the relative risks of cancer and fracture from HRT. And 25%. Increase. That’s bad.
What was the study? On “16 608, patients, there were more invasive breast cancers compared with placebo (385 cases [0.42% per year] vs 293 cases [0.34% per year]…and the estrogen group had higher mortality (25 deaths [0.03% per year] vs 12 deaths [0.01% per year].”
That is bad. Equally bad was the way the author presented the data, the same author who complains in the opening paragraphs about complex data being presented as looking “superficially adequate to the unsophisticated reader,” but I know when someone is presenting information in an manipulating manner designed to blow smoke out a usually inaccessible area.
In a section worthy of the Vaccine Council or Dr. Mercola, it sounds like people deliberately ignored cancer risk to push estrogen to kill women. Someone mention hype? I know it is important to make a point, but those who were investigating HRT and prescribing it, as I did once upon a time, were doing it carefully and with knowledge that there could be risks.
Information does not exist in a vacuum. When talking with my patients and Mom in the late 1980′s, I basically said, based on the odds, how do you want to live your life?
Lifetime risk is a useful way to estimate and compare the risk of various conditions. Hip fractures, Colles’ fractures, and coronary heart disease, and breast and endometrial cancers are important conditions in postmenopausal women that might be influenced by the use of hormone replacement therapy. We used population-based data to estimate a woman’s lifetime risk of suffering a hip, Colles’, or vertebral fracture and her risk of dying of coronary heart disease. A 50-year-old white woman has a 16% risk of suffering a hip fracture, a 15% risk of suffering a Colles’ fracture, and a 32% risk of suffering a vertebral fracture during her remaining lifetime. These risks exceed her risk of developing breast or endometrial cancer. She has a 31% risk of dying of coronary heart disease, which is about 10 times greater than her risk of dying of hip fractures or breast cancer. These lifetime risks provide a useful description of the comparative risks of conditions that might be influenced by postmenopausal hormone therapy.
That was the kind of information and conversations about HRT I was having with patients in my clinic as I completed my residency, the years the author was at Merck developing drugs. Many patients were far more worried about the disability and pain of fractures than they were of breast cancer.
In continued hyperbole that is totally disconnected from what I remember, he calls HRT a “flagrant example of the harm done by straying from the principles of hypothetical/deductive approach and sound clinical science.”
Really? Did this guy ever take care of patients? Has he ever had to make decisions based on incomplete information? We are only into number two of seven and he last lost me with the hysteria. I wonder how he would suggest exploring the effects of waning estrogen on the health of women? Jump straight to a large trial? Do no preliminary work? Ignore any potential leads? What is the alternative to the incremental, and sometimes erroneous, results of medical understanding? How about fluoride and tooth decay? So many insights start with a guess and a little epidemiology. Sometimes it pans out, sometimes it doesn’t. But you do not know unless you try.
3) if small dosages of vitamins are good for humans, very large doses would be better for everyone.
He then notes the studies that show the hypothesis was wrong. But this was only known after the fact, after the studies, and perhaps using vitamins like drugs would have beneficial effects.
Then the odd summary: ” megavitamin therapy tested in properly controlled trials either does nothing or is harmful (except in a few well defined exceptions).”
So it does nothing except when it does. And how would we know the well defined exceptions unless we did the trials?
He goes from complaining about the science to complaining about the regulatory and commercial issues of megavitamins, changing arguments in midstream. Is it the science or how the science is used? Two different issues.
This is getting tedious, even for me. I will soldier on, although the re re re reading 7 is increasingly painful. The closer I read it, the greater the errors and manipulations; a Mandelbrot set of manipulative medical writing. Soon I will find the indefinite articles and pronouns suspect. I try to skim the Health Ranger for a few chuckles; that is not why I read SI. And when is their swimsuit issue? Oh. Wrong SI.
4) Screening tests beyond the standard medical examination are necessary for identifying disease and the risk of disease in apparently healthy, asymptomatic adults.
I will leave this issue to the more knowledgeable hands of Dr. Gorski. His argument seems to be based on the 20:20 vision of hindsight, which is apparently the primary argument in all seven cases. We thought screening would be effective, studies showed it wasn’t, so the hypothesis was flawed and we should not have suggested screening or done the studies.
The author does not show in this, or other examples, why the ideas were wrong in the context of time the ideas were first offered. It is only viewed through the all powerful retrospectoscope that the author finds his deadly hypothesis. It is ever so easy to predict the past.
He also seems to argue that since our understanding of the ramifications of screening are not perfect, they are suspect, referencing himself for issues with PSA and mammograms (1). The author argues in part that since our understanding is imperfect, it is a deadly hypothesis. I have always been comfortable with making decisions based on incomplete information, as that is the only kind of clinical information we ever have, save for the results of the occasional autopsy. The perfect always being the enemy of the good.
He also complains about genetic screening. He notes that few people with high risk genes will develop disease and they can’t do anything about it, so why bother? I wonder if the author has had much direct patient care. What most patients dislike is uncertainty about the why of their disease and most prefer as much understanding and certainty about their health as they can gather. That is why they bother. And todays why bother may be tomorrows critical insight. I have discussed how the show Connections made an impact on my view of the serendipity underlying advances. It may not be cost effective or useful currently, the author does note that for some patients (breast cancer) it may have utility. Again, it is a deadly hypothesis except when it isn’t. So much sound and fury.
But how do you know until after you have done all the studies and see what works and what doesn’t? His argument still seems to be since in some patients genetic testing has been shown to be of no utility, in the past they should not have done the work to show it is not useful. Except where it is. Sort of like going back in time to kill Hitler as a child because he was found to be evil in the future, even though you could not tell that the babe in the crib was going to be the source of Goodwin’s law. And far worse.
Circular argument much?
I do not get the impression the author is one for thinking outside the box. Usually new ideas lead nowhere, but again, you never know unless you try. Nothing ventured, nothing gained vrs nothing ventured, nothing lost. It is often not the results of studies that are the issue, but how they are portrayed in the media, as noted by the author, and, probably not intentionally, his entire article is a superb example of just that concept. Maybe 7 is really meta.
5) Manipulating one’s nutrition can prevent cancer.
As he says, “In retrospect, this hypothesis does not seem plausible.”
The whole crux of almost every one of his arguments. Repeat after me. In retrospect. In retrospect. In retrospect. In retrospect everything is clear. I have had MD after my name for 27 years, and I remember the uncertainty and interest in all his 7 mostly not so deadly hypotheses. In the beginning, it was not so clear as he makes it out to be. The past is easy to predict.
6) Personalized medicine will greatly advance medical care.
His argument is the same: it hasn’t worked except where it has.
“Personalized medicine has only been shown to be cost effective in a few well defined situations.”
How did we find these well defined situations? Doing a ton of studies that show benefit in some cases and none in others.
I think the solution to this problem is being able to see the future and know in advance which research ideas will bear fruit and which will be a bust. Precognition is apparently the only solution. Miss Cleo may be available to help review research proposals, I understand that her readin’ is free.
7) cancer chemotherapy has been a major medical advance.
Of course, in some cases it has been extremely effective, but the war on cancer has not been what it was promised. Again it seems his argument is the same hindsight argument: when cancer therapy has been effective, it is great, and when it is not so good, we should not have done the work to show that wasn’t effective. Again, I leave the details to Dr. Gorski should he choose to cover the topic.
And of course the author doesn’t have a dog in the fight (and there are those quotes, so commonly used by the dispassionate):
“When one dispassionately weighs the minimal prolongation of ‘good’ life in patients with metastatic cancer versus the very distressing side effects of chemotherapy with ‘targeted’ drugs, the case is close.”
I’m convinced, He is dispassionate. And Jenny isn’t anti-vax, just pro-safe vaccine. Here is my hypothesis to be tested. Anyone who argues they are dispassionate isn’t. They are fooling themselves and trying to fool others with their alleged practice of arei’mnu. Me? I am never dispassionate; although sometimes I do not care, but there is a difference.
Some of his conclusions are reasonable: we need to do our science as best as we can.
The author argues that all these errors and expenditures of his 7 mostly no so deadly hypotheses could have “been avoided if the hypothetical/deductive method had been applied rigorously.” I am not convinced, since most of his arguments are based after the fact. I would be far more impressed if, by using only the hypothetical/deductive approach (no epidemiology, no early studies, no preliminary clinical data, no basic science) if he would predict 7 hypotheses that warrant jumping straight to large, randomized, placebo controlled clinical trials so beloved by the FDA. The Randi prize awaits.
We all need that god like perfection and prescience, unlike those
“guilty of perpetuating worthless practices include “scientists” who repeatedly employ flawed methods and then publish them, government agencies who fund such practices, editors of journals that publish pseudoscience, the USDA and NCI bodies that perpetuate unscientific regimens…”
My. God. The Health Ranger was right. The conspiracy has incorporated itself into every aspect of the Medical-Industrial complex. A different conspiracy than the one we get from the woo world, but everyone is involved.
Putting scientists in quotes. A very Health Ranger thing to do. I don’t suppose he is referring to the “scientists” at Merck who repeatedly employed flawed methods and then published them.
“Approximately 250 documents were relevant to our review. For the publication of clinical trials, documents were found describing Merck employees working either independently or in collaboration with medical publishing companies to prepare manuscripts and subsequently recruiting external, academically affiliated investigators to be authors. Recruited authors were frequently placed in the first and second positions of the authorship list. For the publication of scientific review papers, documents were found describing Merck marketing employees developing plans for manuscripts, contracting with medical publishing companies to ghostwrite manuscripts, and recruiting external, academically affiliated investigators to be authors. Recruited authors were commonly the sole author on the manuscript and offered honoraria for their participation…
This case-study review of industry documents demonstrates that clinical trial manuscripts related to rofecoxib were authored by sponsor employees but often attributed first authorship to academically affiliated investigators who did not always disclose industry financial support. Review manuscripts were often prepared by unacknowledged authors and subsequently attributed authorship to academically affiliated investigators who often did not disclose industry financial support.”
I see people doing the best they can with the tools at hand. Mostly honest people (I say mostly not knowing what their IRS forms show), working within many limitations, to advance medical understanding. They do not deserve quotes applied to their work or the title of pseudoscience. Not everyone is able to achieve the peerless, perfect knowledge bestowed on a Professor of Medicine and Merck Vice President.
We need “honest” corporations. Ironic from a former Merck executive; casting the first stone and all that. I do not need quotes to show my snotty superiority. We need better regulation of “unsafe and unproven products.” Like Merck’s Vioxx?. Ohhh, snap. The Merck shots are cheap shots, I know. But they made me laugh, and above all I like to make me laugh. It is all about me.
Like the Health Ranger, I see someone with a bee in their bonnet, selectively and histrionically arguing in circles, hoping that if the same cognitive errors and circular reasoning are repeated they will be believed as fact. I am not enthusiastic about the conclusions and arguments used, being significantly more flawed than the research he rails against. It is not far in style and content from being in the Natural News. Science, at least, is ultimately self correcting. This article, probably not so much.
Of course, I am nobody from nowhere. Not a professor or scientist or a vice president. I am a clinician and citizen who has to trust his sources of information. I was raised to judge a man by the company he keeps. When the NEJM published garbage on acupuncture, my trust in the Journal fell a notch. The Lancet has always had a reputation of being flaky, it is part of the British charm and I have never held it against them; I just factor it in when reading a paper. The Annals of Internal Medicine has been untrustworthy for years. Clinical Infectious Diseases remains unsullied. Now the Skeptical Enquirer (sic) has slipped a bit as well. 7 was primarily deadly for my confidence in its editors. Oh well, at least I can still trust the material published by DC.
Rationalizations
(1) Crislip et. al. I said it here before, so it must be right.
(2) And if you try sometime you find/You get what you need.

What the heck happened to The BMJ? (2023 version)
The BMJ, once a bastion of evidence-based medicine, has become disturbingly susceptible to publishing biased “investigations” that feed antivax narratives. Its latest report on VAERS by Jennifer Block, who in the past has defended Gwyneth Paltrow and Goop and whose history is not one of supporting science, is just another example of this deterioration.
David Gorski on November 20, 2023
Link

I’ll start this post by noting, as I did the last time I wrote about this topic in a major way, that The British Medical Journal (now The BMJ) is one of the oldest and most respected medical journals in the world that also has a history of publishing some excellent investigative journalism. For instance, well over a decade ago, it was The BMJ that published investigative journalist Brian Deer’s reports recounting the strong evidence indicating that Andrew Wakefield’s case series published in The Lancet in 1998 that linked the measles-mumps-rubella (MMR) vaccine to autism had been based on fraud. As most of our readers know, Wakefield’s study launched the modern iteration of the antivaccine movement and is, as I like to say, the study that launched a thousand quackeries directed at autistic children to “cure” them of their “vaccine-induced autism.” The headline of Deer’s accompanying commentary after his account of how the “case against the MMR was fixed” even referred to Wakefield’s study as “Piltdown medicine“, after the infamous Piltdown Man fraud.
Unfortunately, in more recent years, the “investigative journalism” published by The BMJ has left much to be desired. Let’s just say that The BMJ‘s reporting has taken on a distinctly credulous bent when it comes to antivaccine narratives. The prominent example that I wrote about two years ago was conspiracy-mongering free lance “journalist” Paul Thacker‘s report, “Covid-19: Researcher blows the whistle on data integrity issues in Pfizer’s vaccine trial,“ At the time, I (and others) dissected all the ways in which this report was misleading, asking (the first time) What the heck happened to The BMJ? Unfortunately, the editors of The BMJ not only extrapolated from a story about supposed lapses at one study site for the Pfizer COVID-19 phase 3 clinical trial to generalize them to the whole clinical trial, but to all industry-supported clinical trials as well, and, when called out on it, doubled down. Tellingly, when Lead Stories published a fact check used by Facebook regarding how misleading the story was, instead of addressing the criticisms, the editors of The BMJ instead wrote an open letter to Mark Zuckerberg, CEO of Meta (Facebook’s parent company) complaining without actually refuting any of the criticisms made by Lead Stories, all while ignoring another story published by MedPage Today recounting many of the same obvious problems with Thacker’s report that Lead Stories had.
Well, here we go again, as last week The BMJ published a misleading report about the Vaccine Adverse Events Reporting System (VAERS) that relies on anecdotes and leaves out critical context in order to produce a picture of a “totally broken” system that is hiding data and intentionally covering up reports of injuries due to COVID-19 vaccines.

Jennifer Block in The BMJ: VAERS is “broken” (and—nudge nudge, wink, wink—therefore COVID-19 vaccines are dangerous)
Last week, The BMJ published a report by “investigative journalist” Jennifer Block about VAERS:
A US reporting system designed to detect potential safety issues with vaccines is supposed to be user friendly, responsive, and transparent. But an investigation published by The BMJ finds it’s not meeting its own standardshttps://t.co/qxH5OzC5YJ
— The BMJ (@bmj_latest) November 10, 2023
Predictably, antivaxxers are amplifying the story and claiming that it’s evidence that they were “right all along” about VAERS:
https://t.co/VNB40ODf46 https://t.co/zeOmRT3vuV
— Old School Tin-Foil-Hat Rock Chick (@KathleenDoty) November 17, 2023
Three years too late and barely scratching the surface. https://t.co/ca7eRQMhkT
— Anne Rose (@MetriMommy) November 17, 2023
OpenVAERS raise concerns about this deceptive bookkeeping system by the CDC.
Vax injured & killed are significantly higher than VAERS reports to the public. Why are they deceiving people?https://t.co/JfGs95EJPu pic.twitter.com/Jc6Yj1JGko
— GQ4U (@GQ4U52) November 18, 2023
Notice that last one, with the meme. Basically, it’s an antivaxxer using Block’s report to argue that OpenVAERS, the fake VAERS database created in 2021 by an antivaxxer named Liz Willner, who scrapes and misrepresents data VAERS data regarding harms produced by COVID-19 vaccines in order to spin a false narrative that vaccine injuries are being “covered up,” was “right all along.” How does her report accomplish this? Let’s dig in.
Here’s anecdote number one:
Three weeks after receiving a second dose of a covid vaccine, Robert Sullivan collapsed at home on his treadmill. An anaesthesiologist in Maryland, USA, he was a particularly fit 49 year old: the week before falling ill, he’d been happily skiing at altitude in Colorado.
Sullivan was given a diagnosis of sudden onset pulmonary hypertension, which is generally progressive, can be fatal, and in most cases can’t be cured. The condition is rare, especially in middle aged men. Sullivan decided to file a report in the Vaccine Adverse Event Reporting System (VAERS), which collects reports of symptoms, diagnoses, hospital admissions, and deaths after vaccination for the purpose of capturing post-market safety signals.
But the submission process was a glitchy race against the clock. “The format is cumbersome and it times you out,” he tells The BMJ. For his troubles, Sullivan received a confirmation by email and a temporary “e-report” number. He learnt from his doctor’s office that a VAERS representative had requested medical records. Then he didn’t hear back for a year.
This is what Block leads with? After all, one would think that a journalist would lead with her strongest anecdote supporting the findings of her report. This is some seriously thin gruel so far. Think about it. What do we have so far? It sounds as though the server was overloaded, but that Dr. Sullivan did manage to make his report. Then, the system appears, thus far, to have worked as intended, even if glitchy, with a VAERS representative contacting his physician for medical records, before no updates for a year. What does this mean? If you believe Block and The BMJ it means:
VAERS is supposed to be user friendly, responsive, and transparent. However, investigations by The BMJ have uncovered that it’s not meeting its own standards. Not only have staffing levels failed to keep pace with the unprecedented number of reports since the rollout of covid vaccines but there are signs that the system is overwhelmed, reports aren’t being followed up, and signals are being missed.
I doubt that anyone would deny that, early after the rollout of COVID-19 vaccines, the system was overwhelmed. After all, this was the largest and fasted rollout of a vaccine since VAERS was created over three decades ago. It was also the first time that the government had so heavily advertised VAERS with each dose of COVID-19 vaccine and encouraged people to report any adverse events (AEs) that they thought to be due to the vaccine. The CDC even created another system (V-Safe) that, unlike VAERS, a passive reporting system, send periodic text messages to vaccinees who signed up to ask them how they were doing. This provided more cues to report to VAERS than had ever been provided before. Now, don’t get me wrong. If the system was overwhelmed and understaffed to the point where its servers would time out and its staff couldn’t follow up in a timely fashion on AE reports, then that should have been fixed.
However, were safety signals being missed? Given how fast safety signals for unusual blood clots, myocarditis, and other issues were identified, studied, and reported, I would argue that VAERS, for all its struggles and shortcomings, did a surprisingly good job serving its intended purpose: To serve as the “canary in the coal mine” to identify possible problems with vaccines meriting further study to determine if they were related to the vaccine. Remember, VAERS cannot demonstrate causation, nor can it accurately estimate prevalence of specific AEs, as I have written many times before dating back to 2006 at my not-so-super secret other blog, where I discussed how antivax lawyers had encouraged reports to VAERS in order to game the database in favor of supporting a causal link between vaccines and autism. Unsurprisingly, with the massive rollout of COVID-19 vaccines, it took very little time at all before antivaxxers were misleadingly claiming that VAERS supported all sorts of AEs due to the vaccines and even massive numbers of deaths, all starting less than two months after emergency use authorization (EUA) for the vaccines.
Let’s see what Block and The BMJ base their conclusions on:
The BMJ has spoken to more than a dozen people, including physicians and a state medical examiner, who have filed VAERS reports of a serious nature on behalf of themselves or patients and were never contacted by clinical reviewers or were contacted months later.
Our investigation has also found that, in stark contrast to the US government’s handling of adverse reaction reports on drugs and devices, the publicly accessible VAERS database on vaccines includes only initial reports, while case updates and corrections are kept on a separate, back end system. Officials told The BMJ that this was to protect patient confidentiality—but this means that patients, doctors, and other public users of the database have access only to an incomplete and uncorrected version.
Sounds pretty damning, doesn’t it? But does the report really support such conclusions? Spoiler alert: This investigation does suggest areas where VAERS has fallen short and could be improved, but not the claim that the CDC and FDA (which run the database) has been intentionally hiding or “covering up” AEs or that it’s missing huge safety signals.
First, though, let’s start with this:
Co-managed by the US Centers for Disease Control and Prevention (CDC) and the Food and Drug Administration, the VAERS reporting system relies on a mixture of voluntary adverse event reports from doctors and patients and mandatory reporting from vaccine manufacturers, which are required by law to report all adverse events they become aware of.
This is an interesting bit of misdirection. It is true that manufacturers are required to report all serious AEs and deaths that they become aware of to VAERS. It is also true, but not mentioned, that drug and device manufacturers are also required to report serious AEs and deaths to the FDA under postmarketing reporting requirements for drugs and medical devices. In other words, VAERS is not the be-all and end-all for reporting AEs, at least not for manufacturers. There are various other pharmacovigilance systems in place to monitor potential AEs. It turns out that the same is actually true for vaccines as well in that there are multiple active surveillance systems, such as the Vaccine Safety Datalink (VSD), the Clinical Immunization Safety Assessment (CISA) project, or FDA’s Post-licensure Rapid Immunization Safety Monitoring System (PRISM). Systems like these actively search for AEs in massive medical record databases that might be linked to vaccination. VAERS, as I often point out, is intended as a hypothesis-generating reporting system, given that reports to it are voluntary (e.g., passive in that it waits for reports instead of, like for instance the VSD, going out an looking for AEs and trying to correlate them to vaccinations), not a system that can prove causality.
In fairness, before I get to the misdirections and malinformation, let’s look at what Block reports that suggests a real problem:
VAERS’s standard operating procedure for covid-19 states that reports must be processed quickly, within days of receipt.12 “Serious reports” trigger the requisition of medical records and at minimum a “manual review,” while deaths and other “adverse events of special interest” may undergo a more “in-depth” clinical review by CDC staff.
However, The BMJ has learnt that in the face of an unprecedented 1.7 million reports since the rollout of covid vaccines, VAERS’s staffing was likely not commensurate with the demands of reviewing the serious reports submitted, including reports of death. While other countries have acknowledged deaths that were “likely” or “probably” related to mRNA vaccination, the CDC—which says that it has reviewed nearly 20 000 preliminary reports of death using VAERS (far more than other countries)—has not acknowledged a single death linked to mRNA vaccines.
Before the pandemic VAERS was receiving nearly 60 000 adverse event reports each year. A 2015 CDC article suggests that the agency had the capacity to request records for just a few thousand serious reports each year.3 But in 2021 the total number of reports shot up to a million, and another 660 000 have been filed since. Nearly one in five meet the criteria of serious. This surge reflects the unprecedented campaign to vaccinate against covid-19—in the US alone some 675 million doses have been administered—and the vast majority of recent reports are related to covid vaccines. The CDC states that, “in the event of a significant increase” in VAERS reports warranting clinical review, the standard operating procedure requires additional CDC Immunization Safety Office staff to process cases.
It is, of course, unsurprising that a system might buckle when it suddenly receives nearly 20-fold more reports than it is used to processing. Moreover, it is now clear that, in retrospect, prior to the pandemic the CDC and FDA were seriously underfunded compared to what their mandate according to law was. The report mentions a Pfizer report from February 2021 that described how the company had onboarded 600 additional full time employees to handle the volume of AE reports and at the time planned to employ a total of 1,800 for this purpose, although the number now is unknown. Block cites:
Freedom of Information Act documents seen by The BMJ suggest that Pfizer has around 1000 more full time employees working on vaccine surveillance than the CDC, despite the latter’s responsibility for handling adverse event reports on all products. The CDC didn’t provide The BMJ with specific numbers, instead stating that its staffing “reflects the needs of the office” at any given time and can range from “several dozen to hundreds,” including contractors and individuals reassigned “from across the agency.” The latest confirmed number of staff, as of November 2022—at the Immunization Safety Office, which houses VAERS—is 70-80 full time equivalent workers.
The wag in me can’t help but point out that this observation suggests that, far from the antivax narrative that portrays big pharmaceutical as being relatively unconcerned about pharmacovigilance and AE monitoring, pharmaceutical companies, for all their flaws and greed, actually do care if their products are hurting people enough to devote considerable resources to the task of postmarketing surveillance for AEs due to their products. Let’s look into more of the specifics, though. On what evidence does Block base her claim that VAERS is “broken”?

A dozen anecdotes, or: Where’s the beef?
So how does Block come to the conclusion that VAERS is “broken”? The first piece of evidence that she cites comes from “more than a dozen” VAERS users who had difficulty filing reports. I kid you not. “More than a dozen.” Whenever I see someone cite more than a dozen, I suspect that what they really mean is thirteen, which is the lowest number that is “more than a dozen,” but, then, I’ve seen this sort of reporting before, where a small number of anecdotes are generalized to be representative of a much larger problem. In fairness, maybe these “more than a dozen” anecdotes are representative of larger problems, but if you’re going to make that argument you’d better have the goods to back it up. I also note that, of the “more than a dozen” anecdotes only a handful are discussed, starting with Dr. Sullivan’s anecdote above (which, tellingly, is never mentioned again after the introduction) and continuing to:
Those people include Patrick Whelan, a rheumatologist and researcher at the University of California Los Angeles, who in 2022 reported how one of his patients, a 7 year old boy, had a cardiac arrest after covid vaccination. The patient was intubated when Whelan filed a VAERS report, and he expected a prompt follow-up call from a CDC investigator.
“I assumed that, since it was a catastrophic event, the safety committee would want to hear about it right away,” he says. But, to his knowledge, nobody called or requested medical records. In an email sent to Whelan months later the FDA said that it had followed up “soon after” receiving his report and had made “several requests” for medical records. The agency added, “Generally speaking, staff might not reach out to providers unless they have specific questions about a case or a VAERS report.”
Wait a minute. So VAERS personnel did follow up and make requests for medical records “soon after” the report? The implication in Block’s reporting is that because Dr. Whelan didn’t receive the request that no request was made, but in reality it would be the hospital records that VAERS would most likely have requested. Moreover, Dr. Whelan’s name sounded familiar, and all I had to do to remember who he was was to search this very blog, where I found that he had been one of the co-authors on Joseph Fraiman’s p-hacking masterpiece of a crap study—which started making the antivax rounds first as a preprint—looking at COVID-19 “serious adverse events of special interest” (SAESIs). The study claimed to be a “reanalysis” of the Pfizer and Moderna phase 3 trials finding that there were far more SAESIs than reported. It didn’t.
It turns out that just searching my blogs for names mentioned here shows that many of the “experts” interviewed by Block have a definite point of view that is biased against COVID-19 vaccines. Indeed, as early as December 8, 2020 (the meeting after which the FDA issued the EUA for the Pfizer vaccine—Dr. Whelan expressed extreme concern about using the vaccine in children because of systemic inflammation and toxicity due to spike protein. This was before the vaccine was even being considered for use in children, and he has been quoted saying things like, “It is a lot easier to put shot in arms than test people, than await results and then give people a choice.”
Later in the story, this case is used to build the conspiracy theory that the CDC is “covering up” COVID-19 vaccine injury and deaths, although before we get to that, there is another familiar story:
James Gill has been a medical examiner and forensic pathologist for 25 years and is currently chief medical examiner for the state of Connecticut. In June 2021 he made the first VAERS report of his career. It was for a 15 year old boy who died suddenly days after getting a second jab—what Gill concluded on autopsy was “stress cardiomyopathy following second dose of the Pfizer-BioNTech covid-19 vaccine.”567
Gill, who has appointments at Yale University and the University of Connecticut, can’t recall getting any calls from VAERS after he filled out the online form, and he still has only a temporary “e-report” number. After he published the case reports in the Archives of Pathology & Laboratory Medicine in February 2022, however, the CDC did respond—in the form of a letter to the editor contesting Gill’s findings.567
I link to that letter regarding his report cited above and cite part of it, because the CDC did more than contest Dr. Gill’s findings. It brought home the evidence:
The conclusion of IDPB, based on the composite histologic, immunohistochemical, and molecular findings, was death attributable directly to C septicum sepsis. This conclusion is supported further by several salient and well-recognized clinicopathologic characteristics of this disease. Clostridium septicum sepsis is characteristically a fatal and fulminating infection that often presents with nonspecific signs and symptoms. The infection is lethal in approximately 60% to 70% of cases, and death typically occurs within 12 to 48 hours of symptom onset.7–9 In contrast to the report by Gill et al,1 information reported to the Vaccine Adverse Event Reporting System10 noted that the patient described “flu-like symptoms” for 2 days before death. Clostridium septicum produces multiple toxins that cause necrosis of striated muscle cells9,11 and inhibit influx of neutrophils to infected tissues; indeed, paucity of neutrophilic infiltrates in tissues infected with C septicum is considered a hallmark of this disease.9,12 Clostridium septicum is not considered normal flora of the human intestinal tract,13,14 but rather an opportunistic invader of immunologically compromised hosts, particularly persons with colonic adenocarcinoma, leukemia, diabetes, bowel ischemia, or cyclic, congenital, or acquired neutropenia.7,8 Spontaneous infections have been described for a few pediatric patients with no recognized risk factor and for whom microscopic breaches in the mucosa of the large intestine were considered the likely portal of entry.8,15 No representative samples of the small or large intestine were provided to the IDPB for evaluation; however, histologic evidence of bacterial invasion of the external surfaces of the adrenals, kidneys, liver, and spleen support an intraabdominal source of infection. The IDPB’s initial results were reported to the submitters on August 19, 2021, and an addendum report was issued on October 22, 2021. Of these findings, only the negative SARS-CoV-2 RT-PCR result is included in the paper.
The report by Gill et al1 unfortunately omitted many important findings provided by pathologic evaluations at the CDC that suggested an alternate cause of death for patient A and identified a specific infectious cause of death for patient B. These omissions could lead incorrectly to the assumption that COVID-19 vaccines were directly responsible for the deaths of these 2 patients. We believe that providing these important pathologic findings will allow readers a fuller perspective of the causes of death in these cases.
Again, this is more than just “contesting” Dr. Gill’s findings. It’s investigating the VAERS report in depth and the cause of death. It is the CDC doing its job. You can argue that it should have done this before Dr. Gill published his report. That could indeed be viewed as a failure. However, this investigation certainly does not support Block’s conclusion that the CDC fails to follow up. Moreover, the real burn comes where the CDC mentions that it had reported all this to Dr. Gill before he had published his report but that Dr. Gill had left out this critical information. No wonder Block just breezed over this as the CDC merely “contesting” Dr. Gill’s conclusions.
Then, of course, there’s the group of “physicians and advocates” that met with FDA officials in 2021 and 2022 about VAERS:
A group of physicians and advocates have met multiple times with FDA representatives from 2021 to 2022, including Peter Marks, director of the Center for Biologics Evaluation and Research, and Narayan Nair, the FDA division director who oversees VAERS, to express their concerns that the system isn’t operating as intended and that signals are being missed.
One participant in the group—Helen, an intensive care and emergency physician who asked to use a pseudonym to protect her employment—had filed reports on behalf of several patients, including six who died, among them a 40 year old man. She tells The BMJ that she received a request for medical records for just one of the deaths and for two of her patients admitted to hospital. “You’re not meeting your defined definition of follow-up,” she told Nair and Marks during an online meeting on 22 March last year. “There’s a breakdown in your system.”
The BMJ obtained audio of these meetings, totalling seven. In March 2022, in response to the physician, Nair responded that if a report meets the definition of serious, “there really should be a request for records.” In the case of a reported death, “we prioritise getting the records for those extremely quickly.” He added, “You know, we’ve received a large volume [of serious reports]. And I don’t know what the back-up is.”
I like how Block makes it sound as though getting the audio for these meetings was some sort of coup, as the audio files are all over antivax social media. For example, in September Dr. Drew Pinsky posted one of his shows to X that included the recordings, specifically citing one of the meetings as the March 2022 meeting:
The call – recorded by Dr. Joseph Fraiman in March 2022 – included FDA Director Dr. Peter Marks, Dr. Peter Doshi, Dr. Celia Witten, Dr. Sarah Walinsky, Dr. Lorrie McNeill & many others. "This was a top level meeting," says Dr. Fraiman. It "terrified my entire work group."
— Dr. Drew (@drdrew) September 11, 2023
This call was recorded by Dr. @JosephFraiman in March 2022. He played part of the call on the 9/6/23 episode of @AskDrDrew w/ @DrKellyVictory & Dr. @JesslovesMJK. The audio has been enhanced for clarity but is unaltered. Watch the full episode for more: https://t.co/ege2BHiRoj
— Dr. Drew (@drdrew) September 11, 2023
I note that this episode of Dr. Drew’s podcast features Jessica Rose, who is antivax to the core and known for dumpster diving into VAERS, and Dr. Kelly Victory, a COVID-19 crank since 2020 who tried to portray the virus as basically harmless (unless you’re old) but is now fully antivax. Listening to Dr. Fraiman defend his execrable study on SAESIs, I can only picture FDA officials facepalming and eyerolling at these claims, because, fortunately for them, this was a telephone call and not a Zoom meeting. It’s also a straw man to claim, as Dr. Fraiman does, that the FDA is not doing statistical testing on these adverse events. If you listen to the response, what the FDA said is that it doesn’t compare apples and oranges; i.e., rates of different AEs that are unlikely to be due to the same mechanism. In any event, there’s nothing special about this phone call. No doubt Dr. Whelan just gave Block the audio, and the FDA said at one of the meetings:
At the following meeting in May 2022 an FDA spokesperson, Lorrie McNeill, said that “for all of the reports categorised as serious, the VAERS contractor either obtained records, or made several attempts to do so and closed out a report if they didn’t get a response.”
The closest Block comes to citing actual evidence is to point to a “study” by React19, described as an “advocacy group of some 30 000 people who have experienced prolonged illness after covid vaccination.” React19 claims that a large percentage of the 126 reports from its members either never got a permanent VAERS number (22%) or had “had disappeared from the system entirely—in other words, one in three of the reports they looked for didn’t show up in the publicly searchable database.” Of course, this likely suggests that the cases were investigated.
I take whatever React19 reports with a grain of salt, however. If you visit the React19 website, it becomes clear that this is an antivaccine advocacy group. It’s full of “testimonials” blaming vaccines for serious injury and a board with a notable paucity of actual scientists and practicing physicians, which makes me wonder who will peer review applications for their “research grants,” plus a “research database” that just lists URLs to every paper under the sun about COVID-19 adverse reactions, regardless of quality. Its website also features a lot of testimony from lawyers, patients, and people denying that they are antivax. React19 does, however, have its own physician provider network where it recruits doctors to treat “vaccine injury,” something that reminds me very much of the old Defeat Autism Now! (DAN!) antivax doctors back in the day, particularly given the number of acupuncturists, “integrative” and “functional” medicine” doctors, and the like. (Antivax quack Lee Vliet is one.) There’s also a sentence near the end of the list stating, “Many of our networked doctors do not wish to be listed. We can connect you!” (I wonder why they don’t wish to be listed.) Also, I just saw this week that one of the fathers of the children featured in an antivax propaganda movie Shot Dead wears a React19 hat the entire time he’s on camera.
Enter the conspiracy theory.
“Separate systems” or following HIPAA?
Up until this part, Block’s report is mainly anecdotes, quotes, and trying to link the anecdotes to a more general failure of VAERS, the CDC, and the FDA based on insinuation and rather thin gruel. It is when Block states that there are two systems, one public, and one private, and that the private system has all the “real dirt” that is being hidden from the public that we get to a new iteration of what I like to call the central conspiracy theory of the antivax movement: That “They” (the FDA, CDC, medical profession, pharma, etc.) “know” that vaccines are deadly but are covering it up.
Here’s what I mean. She comes back to Dr. Whelan and uses his claimed experience to spin a conspiracy narrative:
A week after Whelan filed the VAERS report for his young patient in Los Angeles, he had cause to update it. The boy’s condition didn’t improve, and the decision was made to terminate life support. But “there was no mechanism for [updating] it,” he tells The BMJ. “The only option I had was to make a new VAERS report.”
Three weeks later he met with the FDA in his capacity as a researcher to discuss a forthcoming paper, and without having planned to he mentioned the case and the lack of follow-up. Nair reached out by email the next day, and six weeks later Whelan discussed the case with Nair and a press officer for around 35 minutes.
Today, however, that VAERS report still shows the child as having been admitted to hospital. “I made the false assumption that that conversation would result in an adjustment in the publicly reported case,” says Whelan. “Think of all the people who are using VAERS data as a means to assess what’s happening with the vaccines—except in this case you’d be left with the false impression that the child had had a serious adverse event,” rather than seeing that he’d died.
The BMJ has found that the FDA and CDC essentially maintain two separate VAERS databases: a public facing database, containing only initial reports; and a private, back end system containing all updates and corrections—such as a formal diagnosis, recovery, or death.
Unfortunately, she is forced to cite an FDA spokesperson as to the reason for the two systems:
In December last year Nair explained the situation at a meeting with advocates. He said, “There’s two parts to VAERS, the front end system and the back end . . . Anything derived from medical records by law” can’t be posted on the public facing system. The CDC has told The BMJ that “protecting patient confidentiality is a priority.”
Damn that pesky HIPAA! It’s also not as though there isn’t a history of antivaxxers trying to use a vaccine safety database to suss out personally identifiable information. Mark and David Geier were busted for doing just that when merging datasets to do one of their antivax studies using the VSD and had their institutional review board (IRB) approval yanked as a result. I would also point out that this “backend VAERS” with investigated and adjudicated reports is what is used when the FDA and CDC present to the Advisory Committee on Immunization Practices (ACIP) periodically.
In fairness, Block notes that the FDA Adverse Event Reporting System (FAERS), which collects reports on drugs, does maintain a publicly accessible database that gets updated, as does the agency’s Medical Device Reporting system, which leads to the question of why VAERS can’t do the same. Perhaps VAERS should, but it would take funding and will to updated VAERS and scrub reports of any patient-identifiable information in order to produce a publicly accessible database with more than just the raw VAERS reports. In addition, in order to test these databases, I did a few searches for various drugs and devices, and I was unsure whether to be relieved or disappointed in the small number of reports. In other words, I’m not sure whether FAERS is so much superior.
That being said, let’s review the structure of this report. First, it lists a handful of anecdotes about people and healthcare professionals who had difficulty entering reports into VAERS, getting followup from VAERS, and/or seeing whether VAERS ever followed up with medical record requests, which could suggest legitimate shortcomings in the system as the grain of truth that all good conspiracy theories need at their core, asking one legitimate question: Why isn’t VAERS maintained and updated better for the public? If that’s all it had done, I would likely have had no problem with it. Vaccine advocates have long complained about VAERS, in particular how easily it is gamed and whether reports are adequately vetted. Everyone remembers the classic example from two decades ago when a vaccine advocate entered a fake report saying that the influenza vaccine had turned him into a creature resembling The Incredible Hulk, a report that was accepted. (In fairness again, VAERS personnel did eventually contact the advocate to inquire about it.)
But Block didn’t confine herself to that, because the narrative is that the shortcomings in VAERS exist in order to downplay or cover up vaccine injuries. So she intersperses these anecdotes with quotes from sources who are at least suspicious of COVID-19 vaccines or have co-authored bad studies that seek to make the vaccines look dangerous, at every step portraying the FDA and CDC as stonewalling her for answers, while cleverly adding a throwaway observation that “if a reporter successfully treated a diagnosis and the patient improved, or if they confirmed that the cause of the illness was unrelated to a vaccine, this wouldn’t be reflected in the public database.” Of course, the publicly facing database doesn’t include the adjudication of each case as being caused by or not caused by a vaccine, just the report that was made, which is why antivaxxers have loved dumpster-diving into VAERS for decades.
Block’s narrative isn’t even fully cohesive, either. There are internal contradictions, given how at one point she notes that “it took just six VAERS reports of thrombosis with thrombocytopenia syndrome after administration of the Janssen covid vaccine for regulators to issue a ‘pause’ in April 2021,” which to me sounds as though the FDA was doing its job. I would also argue that the FDA detected the increased risk of myocarditis in young males in a pretty timely fashion as well. None of that stops her from going into “just asking questions” (JAQing off) mode:
Ralph Edwards, former director of the Uppsala Monitoring Centre and until recently editor in chief of the International Journal of Risk & Safety in Medicine, explains that monitoring systems such as VAERS excel at detecting adverse reactions that occur very soon after vaccination or are known from other vaccines, such as anaphylaxis or Guillain-Barré syndrome. But detecting new and unusual reactions, especially those with latent effects, has been an ongoing challenge in the world of pharmacovigilance.
Edwards tells The BMJ, “If something hasn’t been heard of before, it tends to be ignored.” He explains that regulators may be relying too heavily on epidemiological evidence to acknowledge a signal. VAERS alone is unlikely to capture such long term adverse outcomes unless reports are regularly updated and reviewers are closely following such cases—“a real catch 22,” he says. “You’ll never get the evidence unless you have the idea to look for it in the first place.”
While Edwards is not to my knowledge an antivaxxer, the reason he is being quoted here is because what he is saying, although not unreasonable in and of itself, in the context of the story supports a favorite narrative of antivaxxers, namely that vaccines cause horrific effects that are not seen immediately. In other words, it supports the antivax narrative about virtually every vaccine that it hasn’t been studied long enough to rule out horrible “long term” effects. Never mind that there are no known AEs from vaccines that take longer than six months to manifest themselves and that all known AEs from vaccines (even “delayed” reactions) occur relatively soon after vaccination (as in hours to days to weeks), not many months to years.
Block also cites Svetlana Blitshtey, a neurologist and researcher at the University at Buffalo who believes that vaccines postural orthostatic tachycardia syndrome (POTS). I note that she has served as expert witness in at least one case before the Vaccine Court trying for a case of POTS after HPV vaccination. (POTS is a “vaccine injury” that is most often attributed to HPV vaccines.) I’ve touched on POTS before, noting that large studies have not found good evidence of a link to HPV vaccination, including in VAERS. (Dr. Vincent Iannelli has a more detailed discussion of the evidence showing that vaccines are unlikely to cause POTS, while I note that antivax “researcher” Yehuda Shoenfeld basically made POTS up as a new syndrome out of whole cloth.) In other words, Block’s choice of experts is quite strategic, designed either to portray VAERS staff as unresponsive, VAERS as secretive, or to JAQ off about “long term side effects” that VAERS won’t catch because of the problems a small number of reporting patients and doctors have experienced, adding:
Helen is calling for an end to the “negative feedback loop . . . the FDA is not naming additional adverse reactions to the vaccines because the passive surveillance systems aren’t displaying it. But the passive surveillance systems aren’t displaying it because physicians are blinded to the adverse reactions in their patients, and thus aren’t reporting them.”
I’m really wondering which emergency medicine and critical care physician “Helen” is. I doubt that it’s Kelly Victory, given that Dr. Victory has a lucrative social media grift going with Dr. Drew pushing her antivax message. (It wouldn’t surprise me if it’s a member of the Frontline COVID-19 Critical Care Alliance or some such group.)
Overall, the entire narrative of this story is one of conspiracy theory, in which the FDA and CDC are gaming VAERS to cover up the death and destruction wrought by COVID-19 vaccines and brave heroic truth tellers like these “over a dozen” doctors with anecdotes, an antivaccine advocacy group and antivax investigators, along with some investigators JAQing off are fighting to uncover The Real Story.
The BMJ sinks lower and lower
When I first wrote about how low The BMJ has been descending, it was in the context of its story about the Pfizer vaccine clinical trial written by Paul Thacker. Besides being known for promulgating anti-GMO conspiracy theories, Thacker is a “journalist” who is known for using abusive Freedom of Information Act (FOIA) requests in order to harass scientists, in particular Kevin Folta. A while back, Steve Novella discussed how Thacker attacked him (and me) as “vaccine cheerleaders” based on our supposed denial and downplaying of potential complications due to the Johnson & Johnson COVID-19 vaccines. Thacker was, as usual, peddling nonsense. Before that and continuing to this day, for some inexplicable reason The BMJ has kept Peter Doshi on as a senior editor, despite his long history of playing footsie with the antivax movement. Doshi, of course, has parroted more than a few antivaccine talking points himself while trying to portray himself as an authority on influenza and the flu vaccine. Through it all, he’s played the “victim” card that people who are borderline antivaccine or antivaccine love to play, claiming that they are “just asking questions” and that anyone who “questions” vaccines is labeled, in a knee-jerk fashion, “antivaccine”. In fact, Doshi’s borderline antivaccine stylings go back further than 2009, when senior pseudonymous epidemiologist whose blog I used to read religiously back then chastised him for an “unhelpful” commentary in which Doshi had claimed that estimates for yearly influenza deaths were “grossly inflated.” That was 2006. During the pandemic, he has published in The BMJ articles that falsely claim that the actual efficacy of COVID-19 vaccines in the randomized clinical trials was 19%, rather than over 90%, a claim that antivaxxers ran with to produce misleading claim that the vaccine was only 19% effective, misinformation that I discussed elsewhere, as did others, such as Skeptical Raptor and John Skylar.
But what about Jennifer Block? The name sounded familiar, which led me to do a few searches, where I found that, just as COVID-19 was starting to make the news in early 2020, she had co-authored a screed defending Goop’s Gwyneth Paltrow and her promotion of quackery in the name of “wellness,” characterizing such attacks as due to “patriarchy” and “misogyny.” A bit earlier, she had published the same message in Scientific American, just without Goop:
Embarassing. "But in every edition [the book @DrJenGunter criticizes] cautiously reported some version of 'some of us have had success.' There’s nothing scandalous or unscientific or pseudoscientific about that statement."
THAT IS THE VERY DEFINITION. @tylerblack32 I can't even.
— Ryan Black ⚖🎮🏀🍁 (checkmark) (@RyanJBlack) November 28, 2019
That’s not a great look. A quick search of her X/Twitter feed for the word “vaccine” reveals that she promotes a lot of antivaccine narratives, including false claims that COVID-19 vaccines don’t prevent transmission at all, that “natural immunity” is superior, and that we should embrace “urgency of normal.” Meanwhile, her Twitter feed is a cesspit of transphobia about “gender affirming care.” (Just peruse it for yourself if you don’t believe me.) As for the resemblance of her report to Paul Thacker’s “investigation,” I discovered that she had been Opinions Editor at Grist when Paul Thacker published an article there, but there’s no way to know if they worked together.
What I do know is that The BMJ has become a source of what is known as malinformation. Malinformation differs from misinformation and disinformation in that it is “nominally true information, misleadingly presented or curated with the intention to deceive or mislead.” In my opinion, Thacker’s report from 2021 qualifies in that it found some problems at one site for the Pfizer clinical trial but then exaggerated them and tried to generalize them to the whole trial, all to cast doubt on the safety of the COVID-19 vaccine. Block’s report does the same thing, finding problems with VAERS and implying—never outright arguing directly—without proving that these problems mean that VAERS is completely unreliable and that COVID-19 vaccine safety is in doubt. (And don’t even get me started on how many times Peter Doshi has published dubious antivax-adjacent “analyses” and “investigations.”)
There was a time when The BMJ was not just a great biomedical journal but also did some really good investigative journalism related to science and health, as it did back when Brian Deer blew the lid off of Andrew Wakefield’s MMR fraud in its pages. These days, it’s difficult to trust anything that The BMJ publishes, particularly when it employs muckraking conspiracy mongering hacks like Paul Thacker and Jennifer Block and then champions their work as anything more than muckraking conspiracy mongering malinformation. In fact, I’m now starting to doubt the quality of the scientific studies that it publishes, given how low the quality control bar has fallen for “investigative journalism” and analysis, particularly with respect to COVID-19 and vaccines.

Science denial, overconfidence, and persuasion
The pandemic has brought scientists who have rejected science with respect to COVID-19 public health measures a disturbing level of influence. Recent research suggests reasons why and who among the public susceptible to such misinformation remains persuadable.
David Gorski on August 22, 2022
Link

If there’s one thing that the COVID-19 pandemic has brought into sharp relief that the vast majority of physicians and scientists either didn’t appreciate or downplayed as unimportant is just how easily people can be led into believing in conspiracy theories and science denial such as antivaccine pseudoscience, including highly educated professionals, such as physicians and scientists. In the slightly less than two and a half years since COVID-19 was declared a pandemic, just at this blog alone we’ve written about many examples whom, before the pandemic, we had never heard of, such as Peter McCullough, Michael Yeadon, Robert Malone, Simone Gold, and Geert Vanden Bossche. Others were academics with whom I had not been familiar, such as the Great Barrington Declaration (GBD) authors Martin Kulldorff, Jay Bhattacharya, and Sunetra Gupta, all of whom had been well-respected before their pandemic disinformation heel turn first to COVID-19 contrarianism and then to outright antivaccine disinformation. There were also scientists with whom we were familiar, making their turn to contrarian but whose turn to contrarianism and science denial actually shocked me, such as John Ioannidis, and ones whose heel turn with respect to science surprised me less, such as Vinay Prasad. Throughout the pandemic, I was never surprised when scientists and physicians who had been antivaccine prior to the pandemic (e.g., James Lyons-Weiler, Paul Thomas, Sherri Tenpenny, Andrew Wakefield, and Joe Mercola) immediately pivoted to COVID-19 minimization and antivaccine disinformation weaponized against COVID-19 vaccines, but some of the others genuinely puzzled me with how easily they had “turned” into spreaders of misinformation.
Looking over the list of these people and others, one can find a number of commonalities shared by some, but not all, of the contrarians listed above. . First, most of them have an affinity towards at least libertarian-leaning politics. Second, few of them have any significant expertise in infectious disease or infectious disease epidemiology. One exception is Sunetra Gupta, co-author of the GBD, is a British infectious disease epidemiologist and a professor of theoretical epidemiology at the Department of Zoology, University of Oxford. She’s a modeller, but early in the pandemic her group’s modeling, which concluded that by March 2020 more than half of the UK population might have been infected with the novel coronavirus and therefore “natural herd immunity” was within reach, disagreed markedly with the scientific consensus, which might have played a role. (Instead of wondering where she might have gone wrong, Gupta appears to have doubled down.) However, most of the others above have little or no directly relevant experience in infectious disease, infectious disease epidemiology, virology, immunology, vaccine development, or molecular biology. Yet during the pandemic they became famous—or at least Internet influencers, although a number of them—I’m looking at you, Kulldorff and Bhattacharya—have shown up fairly regularly on Fox News to attack public health interventions and mandates targeting the pandemic.
So what is the common denominator? A recent study published in ScienceAdvances last month suggests at least one reason. Steve Novella wrote about it on his personal blog the week that it was published, but I wanted to add my take and integrate it into a more general discussion of how people—including some who, one would think, should know better based on their education and profession—can so easily fall prey to science denial and conspiracy theories. As Steve noted in his post, for most people other than experts in relevant fields, a very good “first approximation of what is most likely to be true is to understand and follow the consensus of expert scientific opinion.” As he put it, that’s just probability. It doesn’t mean that the experts are always right or that there’s no role for minority—or even fringe±—opinions. Rather, as Steve put it:
It mostly means that non-experts need to have an appropriate level of humility, and at least a basic understanding of the depth of knowledge that exists. I always invite people to consider the topic they know the best, and consider the level of knowledge of the average non-expert. Well, you are that non-expert on every other topic.
It is that humility that is lacking in these people (at least), as suggested by the title of the study, Knowledge overconfidence is associated with anti-consensus views on controversial scientific issues.
Overconfidence versus scientific consensus
The SBM crew has consistently emphasized the need for humility in approaching science. I would add that this need is not limited to science in which one is an expert. For instance, ostensibly I’m an expert in breast cancer, which means that I know an awful lot about the biology of breast cancer and its clinical behavior. However, I am a surgical oncologist, not a medical oncologist; I would not presume to tell my medical oncology colleagues which chemotherapy or targeted therapy drugs they should give my patients after I operate on them, and, I would hope, they don’t presume to tell me who is a surgical candidate, what operation to perform, or how to perform it. Similarly, based on education, experience, and publications, I can be considered an expert in some areas of basic and translational science, such as breast cancer biology related to a certain class of glutamate receptors, tumor angiogenesis, and other areas in which I’ve done research and published.
Finally, after nearly 20 years of studying anti-science misinformation, the antivaccine movement, and conspiracy theories in general, I like to think that I have a certain level of expertise in these areas. That’s why, when COVID-19 first hit, I was careful about making declarations and tended to stick with the scientific consensus unless new evidence suggested that I should do otherwise. However, I did immediately recognize the same sorts of science denial, antivaccine tropes, and conspiracy theories that I had long written about before the pandemic being applied to the pandemic, which is what I wrote about, such as early claims that SARS-CoV-2 is a bioweapon and misuse of the Vaccine Adverse Events Recording System (VAERS) database. The last of these many of us had been predicting before COVID-19 vaccines received emergency use authorizations (EUAs) from the FDA and I wrote about within weeks of the rollout of the vaccines, as antivaxxers were already doing what they had long done for vaccines and autism, miscarriages, and sudden infant death syndrome (SIDS). I recount these incidents mainly to demonstrate how, if you don’t know the sorts of conspiracy theories that have long spread among antivaxxers, you wouldn’t know that everything old is new again, there is nothing new under the sun, and antivaxxers tend just to recycle and repurpose old conspiracy theories to new vaccines, like COVID-19 vaccines. If many of the people above, most of whom self-righteously proclaim themselves “provaccine,” did know these tropes beforehand, I like to think that some of them might not have turned antivax.
Let’s move on to the study, which comes from investigators from Portland State University, the University of Colorado, Brown University, and the University of Kansas. From the title of the study, you might think that this is primarily about the Dunning-Kruger effect, a cognitive bias well known among skeptics in which people wrongly overestimate their knowledge or ability in a specific area. While there are criticisms of the Dunning-Kruger model, it has generally held up pretty well as one potential explanation of how people come to believe anti-consensus views.
This particular study, the authors point one important point out before they describe their methods and results:
Opposition to the scientific consensus has often been attributed to nonexperts’ lack of knowledge, an idea referred to as the “deficit model” (7, 8). According to this view, people lack specific scientific knowledge, allowing attitudes from lay theories, rumors, or uninformed peers to predominate. If only people knew the facts, the deficit model posits, then they would be able to arrive at beliefs more consistent with the science. Proponents of the deficit model attempt to change attitudes through educational interventions and cite survey evidence that typically finds a moderate relation between science literacy and pro-consensus views (9–11). However, education-based interventions to bring the public in line with the scientific consensus have shown little efficacy, casting doubt on the value of the deficit model (12–14). This has led to a broadening of psychological theories that emphasize factors beyond individual knowledge. One such theory, “cultural cognition,” posits that people’s beliefs are shaped more by their cultural values or affiliations, which lead them to selectively take in and interpret information in a way that conforms to their worldviews (15–17). Evidence in support of the cultural cognition model is compelling, but other findings suggest that knowledge is still relevant. Higher levels of education, science literacy, and numeracy have been found to be associated with more polarization between groups on controversial and scientific topics (18–21). Some have suggested that better reasoning ability makes it easier for individuals to deduce their way to the conclusions they already value [(19) but see (22)]. Others have found that scientific knowledge and ideology contribute separately to attitudes (23, 24).
Recently, evidence has emerged, suggesting a potentially important revision to models of the relationship between knowledge and anti-science attitudes: Those with the most extreme anti-consensus views may be the least likely to apprehend the gaps in their knowledge.
This is, as Steve described it, a “super Dunning-Kruger.” The authors, however, describe existing research and what they think that their research contributes to general knowledge thusly:
These findings suggest that knowledge may be related to pro-science attitudes but that subjective knowledge—individuals’ assessments of their own knowledge—may track anti-science attitudes. This is a concern if high subjective knowledge is an impediment to individuals’ openness to new information (30). Mismatches between what individuals actually know (“objective knowledge”) and subjective knowledge are not uncommon (31). People tend to be bad at evaluating how much they know, thinking they understand even simple objects much better than they actually do (32). This is why self-reported understanding decreases after people try to generate mechanistic explanations, and why novices are poorer judges of their talents than experts (33, 34). Here, we explore such knowledge miscalibration as it relates to degree of disagreement with scientific consensus, finding that increasing opposition to the consensus is associated with higher levels of knowledge confidence for several scientific issues but lower levels of actual knowledge. These relationships are correlational, and they should not be interpreted as support for any one theory or model of anti-scientific attitudes. Attitudes like these are most likely driven by a complex interaction of factors, including objective and self-perceived knowledge, as well as community influences. We speculate on some of these mechanisms in the general discussion.
The authors do this through five studies estimating opposition to scientific consensus, as well as objective knowledge and subjective knowledge of these topics:
In studies 1 to 3, we examine seven controversial issues on which there is a substantial scientific consensus: climate change, GM foods, vaccination, nuclear power, homeopathic medicine, evolution, and the Big Bang theory. In studies 4 and 5, we examine attitudes concerning COVID-19. Second, we provide evidence that subjective knowledge of science is meaningfully associated with behavior. When the uninformed claim they understand an issue, it is not just cheap talk, and they are not imagining a set of “alternative facts.” We show that they are willing to bet on their ability to perform well on a test of their knowledge (study 3).
The key part of the study is portrayed in this graph, Figure 1:

The authors note that the relationship between opposition to the scientific consensus and objective knowledge was negative and significant for all episodes other than climate change, while the relationship between opposition and subjective knowledge was positive for all issues other than climate change, Big Bang, and evolution, where the relationship failed to achieve statistical significance.
The authors also noted one interesting effect, specifically that of political polarization, reporting that, for more politically polarized issues, the relation between opposition and objective knowledge is less negative than for less polarized issues and that the relation between opposition and subjective knowledge is less positive. One way that this sort of result might be interpreted is that advocates who take on an anti-consensus view go out of their way to learn background information about the subject that they oppose. However, in this case knowledge doesn’t lead to understanding, but rather to more skilled at motivated reasoning; i.e., picking and choosing information and data that support one’s preexisting beliefs.
Because another issue is that study subjects with different levels of opposition to a scientific consensus might interpret their subjective knowledge differently, noting that “opponents may claim that they understand an issue but acknowledge that their understanding does not reflect the same facts as the scientific community” and that this “could explain the disconnect between their subjective knowledge rating and their ability to answer questions based on accepted scientific facts.” So to test this, the authors developed a measure of knowledge confidence designed to remove this ambiguity by designing a measure of knowledge confidence that incentivized participants to report their genuine beliefs. Specifically, subjects were given an opportunity to earn a bonus payment by betting on their ability to score above average on the objective knowledge questions or to take a smaller guaranteed payout, with betting indicating greater knowledge confidence.
As the authors predicted the results were:
As opposition to the consensus increased, participants were more likely to bet but less likely to score above average on the objective knowledge questions, confirming our predictions. As a consequence, more extreme opponents earned less. Regression analysis revealed that there was a $0.03 reduction in overall pay with each one-unit increase in opposition [t(1169) = −8.47, P
Finally, the authors applied similar methods to questions of whether or not study subjects would take the COVID-19 vaccine (study #4) and attitudes towards COVID-19 public health interventions (study #5). Study #4 replicated the results of previous studies. There was an interesting curveball in study #5, though, questions on how much study participants think that scientists know about COVID-19, and the results were telling:
To validate the main finding, we split the sample into those who rated their own knowledge higher than scientists’ knowledge (28% of the sample) and those who did not. This dichotomous variable was also highly predictive of responses: Those who rated their own knowledge higher than scientists’ were more opposed to virus mitigation policies [M = 3.66 versus M = 2.66, t(692) = −12, P
This could also suggest why certain individuals who self-identified as liberal or progressive have fallen for COVID-19 contrarianism:
Results from five studies show that the people who disagree most with the scientific consensus know less about the relevant issues, but they think they know more. These results suggest that this phenomenon is fairly general, although the relationships were weaker for some more polarized issues, particularly climate change. It is important to note that we document larger mismatches between subjective and objective knowledge among participants who are more opposed to the scientific consensus. Thus, although broadly consistent with the Dunning-Kruger effect and other research on knowledge miscalibration, our findings represent a pattern of relationships that goes beyond overconfidence among the least knowledgeable. However, the data are correlational, and the normal caveats apply.
Before I move on to the general public, personally, I can’t help but wonder if these results have particular relevance in suggesting how scientists and physicians who should know better could come to hold anti-consensus views so strongly. For example, let’s look at the cases of John Ioannidis and Vinay Prasad. John Ioannidis made his name as a “meta-scientist” or critic of science. His publication record covers documentation of deficiencies in the scientific evidence for a broad number of scientific subject areas, ranging from the effect of nutrition on cancer to, well, pretty much all clinical science. Not long after the pandemic hit I started wondering what the heck happened to him and he started abusing science to attack scientists holding consensus views about COVID-19 as “science Kardashians,” I had already started becoming uncomfortable with his attitude as being more knowledgeable about everything, as evidenced by his arguing that the NIH rewards “conformity” and “mediocrity” when awarding research grants. I would speculate—perhaps even argue—that the overconfidence in his own knowledge described in this study had already infected Ioannidis before the pandemic.
Similarly, before the pandemic Prasad had made his name criticizing the evidence base for oncology interventions. (He is an academic medical oncologist.) In particular, he was known for documenting what he referred to as “medical reversals,” when an existing medical practice is shown to be no better than a “lesser” therapy. Both Steve and I discussed this concept at the time and both agreed that, while Prasad’s work produced some useful observations, it also missed a fair amount of nuance. By just before the pandemic, I note that Prasad had taken to criticizing those of us who were combatting pseudoscience in medicine as, in essence, wasting our medical skills, denigrating such activities as being below him, as a pro basketball star “dunking on a 7′ hoop.” As with Ioannidis, I would speculate—perhaps even argue—that the overconfidence in his own knowledge described in this study had already infected Prasad before the pandemic. One could easily speculate that all of the “COVID contrarian doctors” who went anti-consensus—some of whom even became antivaccine— likely shared this overconfidence before the pandemic.
One thing this study doesn’t address, though, that I’ve been wondering about, and that’s the role of social affirmation in reinforcing anti-consensus views and even radicalizing such “contrarian doctors” further into anti-science views. Vinay Prasad and many of the other physicians and scientists I listed have large social media presences and legions of adoring fans. Ioannidis, although he frequently humble brags about not being on social media, does receive lots of invitations to be interviewed in more conventional media from all over the world, and the reasons are his pre-pandemic fame as the most published living scientists and his COVID-19 contrarian views. Then, for many of these doctors, there are financial rewards. A number of them have Substacks in which they monetize their contrarian and science-denying views.
These would be good topics for other studies. In the meantime, let’s move on to the general public that consumes—and is influenced by—their misinformation.
Who is persuadable?
The Science Advances study notes something that skeptics have known for a long time. It isn’t (just) lack of information that drives science denial. It’s more than that, which is why, in and of itself, trying to drive out bad information with good information generally doesn’t work very well:
The findings from these five studies have several important implications for science communicators and policymakers. Given that the most extreme opponents of the scientific consensus tend to be those who are most overconfident in their knowledge, fact-based educational interventions are less likely to be effective for this audience. For instance, The Ad Council conducted one of the largest public education campaigns in history in an effort to convince people to get the COVID-19 vaccine (43). If individuals who hold strong antivaccine beliefs already think that they know all there is to know about vaccination and COVID-19, then the campaign is unlikely to persuade them.
Instead of interventions focused on objective knowledge alone, these findings suggest that focusing on changing individuals’ perceptions of their own knowledge may be a helpful first step. The challenge then becomes finding appropriate ways to convince anti-consensus individuals that they are not as knowledgeable as they think they are.
This is not an easy task. I frequently point out that hard core antivaxxers, for instance, are generally not persuadable. Indeed, I liken their antivaccine views—or other anti-consensus views—to religion or political affiliation, beliefs intrinsic to their identities and every bit as difficult to change as religion or political orientation. In other words, it’s possible to change such views, but the effort required and failure rate are both too high to make these people a target of science communication. As a corollary to this principle, I’ve long argued that it is the fence sitters who are the most likely to have their minds changed, or to be “inoculated” (if you will) against misinformation.
Last week, a study from authors at the Santa Fe Institute was published in the same journal suggesting that I should be rather humble, as I might not be entirely on the right track, in that people do tend to shape their beliefs according to their social networks and existing moral belief systems. The authors note, as I frequently do:
Skepticism toward childhood vaccines and genetically modified food has grown despite scientific evidence of their safety. Beliefs about scientific issues are difficult to change because they are entrenched within many interrelated moral concerns and beliefs about what others think.
Again, the reason many people gravitate to anti-consensus views with respect to specific scientific conclusions (e.g., vaccine safety and effectiveness) involves social moral beliefs and concerns and personal ideology. To determine who is most susceptible to belief change, the authors, Information alone, although a necessary precondition to change minds, is usually not sufficient to change minds. I would also argue that it’s equally important to identify whose minds are changeable with achievable effort.
To boil this research down, the authors looked at two scientific issues, vaccines and genetically modified organisms (GMOs), using this rationale:
In this paper, we consider attitudes toward GM food and childhood vaccines as networks of connected beliefs (7–9). Inspired by statistical physics, we are able to precisely estimate the strength and direction of the belief network’s ties (i.e., connections), as well as the network’s overall interdependence and dissonance. We then use this cognitive network model to predict belief change. Using data from a longitudinal nationally representative study with an educational intervention, we test whether our measure of belief network dissonance can explain under which circumstances individuals are more likely to change their beliefs over time. We also explore how our cognitive model can shed light on the dynamic nature of dissonance reduction that leads to belief change. By combining a unifying predictive model with a longitudinal dataset, we expand upon the strengths of earlier investigations into science communication and belief change dynamics, as we describe in the next paragraphs.
In brief, investigators constructed a cognitive belief network model to predict how the beliefs of a group of almost 1,000 people who were at least somewhat skeptical about the efficacy of genetically modified foods and childhood vaccines would change as the result of an educational intervention. Using a nationally representative longitudinal study of beliefs about GM food and vaccines carried out at four different times over three waves of data collection (once in the first and third waves and twice in the second wave, before and after an intervention). During the second wave, the authors presented subjects with an educational intervention on the safety of GM food and vaccines, quoting reports from the National Academies of Sciences. Participants were divided into five experimental groups for the GM food study and four experimental groups for the study of childhood vaccines, with one control condition in each study in which participants did not receive any intervention. All experimental conditions received the same scientific message about safety with a different framework. The results were then analyzed using the cognitive network model developed, to look for how beliefs would change in response to the educational intervention.
To sum it up, those who had a lot of dissonance in their interwoven network of beliefs were more likely to change their beliefs after viewing the messaging, although not necessarily in accordance with the message. Overall, the authors found that people are driven to lower their cognitive dissonance through belief change. As the authors put it in an interview for their press release in which they commented on how people with little dissonance showed little change in their beliefs after the intervention and those with more dissonance were more likely to show more change:
“For example, if you believe that scientists are inherently trustworthy, but your family and friends tell you that vaccines are unsafe, this is going to create some dissonance in your mind,” van der Does says. “We found that if you were already kind of anti-GM foods or vaccines to begin with, you would just move more towards that direction when presented with new information even if that wasn’t the intention of the intervention.”
What this study suggests is that targeting such people with science communication can be a two-edged sword in that people with high dissonance will try to reduce their dissonance. Unfortunately, reducing that dissonance might not go in the direction that you think it will. They might reduce their dissonance by moving towards accepting scientific consensus with respect to scientific issues that are contentious among the public, such as vaccines and GM foods, or they might move further into conspiracy:
All in all, we found that network dissonance, belief change, and interdependence relate to each other over time, in line with our model assumptions. Interventions aimed at changing people’s beliefs led to a reconfiguration of beliefs that allowed people to move to lower network dissonance states and more consistent belief networks. However, these reconfigurations were not always in line with the objective of the intervention and sometimes even reflected a backlash.
And:
Individuals are motivated to reduce the dissonance between beliefs and reconfigure their beliefs to allow for lower dissonance. Such a reconfiguration can be, but is not necessarily, in line with the aim of the intervention. The direction in which individuals change their beliefs depends not only on the intervention but also on the easiest way for individuals to reduce their dissonance. This finding also goes beyond the classic finding that inducing dissonance leads to belief change (41–43) by showing that providing individuals with new information interacts with dissonances in their belief network. Individuals with low dissonance are unlikely to change at all, whereas individuals with high dissonance can change in both directions.
So the real question, if this research holds up, is: How do we identify people with a high degree of dissonance who won’t reduce their dissonance in response to a pro-science message by moving deeper into antiscience beliefs and conspiracy theories? More importantly, how do we craft messages that make it less likely that these people will adjust their anti-consensus beliefs in the direction that we want them to, rather than doubling down? Again, it is clear that those who are the most certain and have the least dissonance are the least likely to change their views in response to science communication.
Putting it all together
How does one put the results of these two studies together and combine them with what is already known? From my perspective, I see a couple of very important takeaway points. First, as Steve has said, humility is key. One is tempted to quote Richard Feynman, who famously said, “The first principle is that you must not fool yourself and you are the easiest person to fool.” My colleagues who “go crank” almost all have forgotten Feynman’s warning.
Indeed, those who follow my Twitter feed might have noticed me saying things like this lately:
Yep. The first step towards becoming a skeptic (and not a pseudoskeptic) is to admit to yourself—and really mean it—that you, too, can fall for these sorts of conspiracy theories and errors of thinkings. https://t.co/dgAs0GitTI
— David Gorski, MD, PhD (@gorskon) August 12, 2022
The moment you think you're immune to conspiracy theories, you're doomed. Unfortunately, a lot of self-identified movement skeptics—some quite prominent in the movement—seem not to have taken these messages to heart, especially lately about, for example, transgender medicine./end
— David Gorski, MD, PhD (@gorskon) August 12, 2022
I won’t name those skeptics here, but the warning above especially applies to people like John Ioannidis, who seems to think that, having spent over a quarter century documenting problems with biomedical research, he is somehow immune to the same errors in thinking that lead others astray. Then there’s Vinay Prasad, who not only doesn’t think that such errors in thinking are important enough to trouble his massive brain with, but has expressed outright contempt towards those of us who do and act upon that belief by combatting medical missinformation. It is not difficult to see how two men who built their academic careers on “meta-science” and, in essence, commenting on deficiencies in science, might come to think that they know better than the scientists with relevant topic-specific expertise, rather than more general expertise about scientific methodology.
Ideology also plays a role. Martin Kulldorff is an excellent example of this. He clearly almost certainly must have had preexisting libertarian views towards collective action led by government as anathema. I say this based on how easily Jeffrey Tucker of the “free market” think tank American Institute for Economic Research enticed him to come to its headquarters, where, having found his people, Kulldorff wanted to hold the conference at which the GBD was birthed more urgently than even Tucker did. Ever since then, Kulldorff, along with his fellow GBD author Jay Bhattacharya, has been slipping further and further from science and deeper and deeper into conspiracy theory and antivaccine advocacy.
As for the rest, most of them weren’t academic, and those who tended to be either adjunct clinical faculty or already flirting with crankdom. It isn’t hard to believe that what drew them into COVID-19 contrarianism was the ego boost from the attention that they got for promoting views that went against the mainstream. I’ll reiterate right here that there is nothing inherently wrong about challenging the mainstream. What matters is that you have the goods—the evidence and/or logic—to back up that challenge. That’s what differentiates legitimate scientists expressing nonmainstream views from the cranks.
Then, of course, the role of grift cannot be underemphasized. As I like to say, ideology is key, but it’s often also always about the grift. Just look at America’s Frontline Doctors if you don’t belief me.
As for reaching the public, what these studies suggest is that science communication is complex and far more difficult than most people appreciate. While it’s undeniably true that the more certain that, for example, an antivaxxer is, the less likely anyone is going to be able to change their mind, predicting who will be persuadable and won’t react to science communication by moving deeper into conspiracy and crafting messages to minimize that possibility are very difficult. The more I learn about this area, the less I feel confident that I understand well. The key, however, is being willing to change one’s views and approach based on new evidence. The application of the findings from these two studies, and many more that not infrequently conflict with each other is the biggest challenge for science communicators. It doesn’t help that we’re in the middle of a pandemic that has resulted in the spread of anti-science disinformation and conspiracy theories, many not innocently spreading but intentionally promoted. That a large amount of the disinformation that we see is not organic, but rather promoted by actors with agendas, just makes the problem even worse.
I don’t mean to finish on low note. The situation is not hopeless. In fact, it is research like this that is most likely to provide the tools to public health officials and science communicators to counter antiscience messages.

One reason mouse studies often don’t translate to humans very well
Mouse models are often used as preclinical models of human disease, but the number of drugs that succeed in mice but go on to be approved as a drug for humans is only about one in ten. A new study comparing gene expression in the cells of human brains with those of mouse brains provides new insight into why.
David Gorski on August 26, 2019
Link

If there’s one thing about science reporting that drives most researchers, particularly medical researchers, bonkers, it’s the hyping of preclinical studies in cell culture and animal models as major breakthroughs, even though the drug or treatment tested has yet to be translated into a treatment for human beings. Don’t get me wrong; sometimes a mouse study can be an important breakthrough. However, far more often than not, treatments that work in mice fail to translate to humans and don’t work or don’t work nearly as well in humans. My favorite example of this is something I studied during my fellowship, angiogenesis inhibitors. Back in the late 1990s, angiostatin and other angiogenesis inhibitors, compounds that could prevent the ingrowth of new blood vessels into tumors, were being touted as a cure for cancer. This was because for all solid tumors at least, cancers rely on the ability to induce the body to generate new blood vessels for their continued growth beyond a couple of millimeters in diameter to feed their voracious needs for oxygen and nutrients. There were spectacular results, too, in mice. My scientific hero, the late great Judah Folkman, could cure cancers in mice. He could even induce tumor dormancy, such that the tumor would shrink to a very tiny size and then remain dormant indefinitely, all using angiostatin and endostatin, two endogenous angiogenesis inhibitors that he had isolated from the urine and blood of mice. Unfortunately, in humans, although angiogenesis inhibitors do work, their effects are nowhere near as dramatic, and they have taken their place as just one type of drug among many used to treat various cancers, with no spectacular results, just incremental improvements in treatment. As Dr. Folkman quipped at the time, “If you have cancer and you are a mouse, we can take good care of you”.
So why do so many treatments that work in mice for various diseases fail to work in humans? For instance, in neuropsychiatric research, the track record of mouse models for predicting the efficacy of new treatments in humans has been particularly abysmal. There was a study published last week that suggests one reason, at least for drugs designed to treat psychiatric and neurological disorders, and that’s previously unappreciated differences in brain function. Before I get into the study itself, let’s let Sharon Begley, writing about the study for STAT News, set the stage:
Lab mice endure a lot for science, but there’s often one (temporary) compensation: near-miraculous recovery from diseases that kill people. Unfortunately, experimental drugs that have cured millions of mice with Alzheimer’s disease or schizophrenia or glioblastoma have cured zero people — reflecting the sad fact that, for many brain disorders, mice are pretty lousy models of how humans will respond to a drug.
Scientists have now discovered a key reason for that mouse-human disconnect, they reported on Wednesday: fundamental differences in the kinds of cells in each species’ cerebral cortex and, especially, in the activity of those cells’ key genes.
In the most detailed taxonomy of the human brain to date, a team of researchers as large as a symphony orchestra sorted brain cells not by their shape and location, as scientists have done for decades, but by what genes they used. Among the key findings: Mouse and human neurons that have been considered to be the same based on such standard classification schemes can have large (tenfold or greater) differences in the expression of genes for such key brain components as neurotransmitter receptors.
I don’t know if the team of researchers really was “as large as a symphony orchestra” (I counted 63 authors, but I didn’t double check; so I could be off by one or two), but it does take a lot of people from several different universities, such as UC-Davis, the University of Washington, Columbia University, UC-San Diego, and multiple institutes) to do a study of this type. In any event, as you can gather from this summary, even though anatomically the neurons and circuits connecting various regions of the brain in mice look very similar to the corresponding neurons and circuits in humans, at the gene expression level they are actually very different.
So here’s what the researchers did. They harvested postmortem brains from male and female donors 18–68 years of age with no known history of neuropsychiatric or neurological conditions as the controls, as well as brain tissue from patients undergoing surgery for epilepsy. From this tissue and from the whole postmortem whole brains, the researchers focused mainly on a part of the brain called the middle temporal gyrus (MTG), which is often available from epilepsy resections and thus allowed the comparison between the tissue of the epileptic subjects and the control subjects. They then subjected these specimens to single-nucleus sequencing:
Single-cell transcriptomics enables molecular classification of cell types, provides a metric for comparative analyses, and is fuelling efforts to understand the complete cellular makeup of the mouse brain18 and even the entire human body19. Single-cell RNA sequencing (scRNA-seq) of mouse cortex demonstrates robust transcriptional signatures of cell types20,21,22 and suggests around 100 cell types per cortical area. Dissociating live cells from human brain is difficult, which makes scRNA-seq challenging to apply to this type of tissue, whereas single-nucleus RNA-seq (snRNA-seq) enables transcriptional profiling of nuclei from frozen human brain specimens23,24. Of note, nuclei contain sufficient gene-expression information to distinguish closely related cell types at a similar resolution to scRNA-seq25,26, but early applications of snRNA-seq to human cortex did not have sufficient depth of coverage to achieve similar resolution to mouse studies27,28. Here, we established robust methods for the classification of cell types in human brain using snRNA-seq and compared cortical cell types to reveal conserved and divergent features of human and mouse cerebral cortex.
To boil it all down, doing single-cell sequencing of the genome would be the best approach, but in the brain apparently separating single cells from the brain without killing them is a technical challenge; so the authors used the next best method, single-nucleus sequencing. The disadvantage is that nucleus doesn’t contain all the messenger RNA (mRNA) being made, as the mRNA is transported out of the nucleus to be translated into protein. However, based on previous research, in this case single-nucleus sequencing is as good as single-cell sequencing for determining the differences in gene expression between cell types in the brain. As for the rest, RNA-Seq is a next generation sequencing technique that allows the sequencing of every mRNA sequence detectable in a given RNA isolate from cells or tissue; it’s become so sensitive that it is now possible to perform RNA-Seq on single-cell and single-nucleus RNA isolates.
Now consider the sheer magnitude of this study. Nuclei were isolated from eight donor brains, with most coming from postmortem donors (n = 15,206) and a minority (n = 722) from layer (L)5 of MTG removed during neurosurgical procedures:
In total, 15,928 nuclei passed quality control, including those from 10,708 excitatory neurons, 4,297 inhibitory neurons and 923 non-neuronal cells. Nuclei from each broad class were iteratively clustered as described26 (see Methods). Clusters were generally robust to different iterative clustering methods and were distinguished from nearest neighbours by at least 30 differentially expressed genes and at least one, and often more, binary markers.
Translation: After sequencing, the researchers noted that the various cell types could be robustly identified by various mathematical clustering methods used to analyze genomics data.
The purpose of this study was to develop and validate methods and a system for using snRNA-Seq to catalog the cell types in the MTG and later elsewhere in the brain. Given that, it’s not important to go through all the details of the cell types and their locations that were documented, especially for a non-neurologist or non-neuroscience geek. A few observations were interesting, though. For example, excitatory neurons were more widely distributed than expected, with most excitatory neuron types not being restricted to single layers. Three types were more common in L2-L3, while ten types were enriched in L3-L6, leading the authors to note that this “heterogeneity implies that anatomical laminar location alone is insufficient to predict neuron type, although it remains to be seen whether this is a feature of MTG or human cortex generally”.
When the authors compared these single-nucleus gene expression profiles to those of cells in the same region of mouse brain, they noted several differences. Indeed, they found that neurons in mice and humans that have been generally considered to be the same based on anatomy, structure, histology, and standard classification schemes can actually have ten-fold or even greater in the expression of genes for very important proteins, such as neurotransmitter receptors. (Neurotransmitters are the peptides that neurons use to communicate with each other.) For example:
Comparison of homologous types showed a mix of conserved and divergent expression. The Sst Chodl type (Inh L3–L6 SST NPY in human) had conserved expression overall, but 18% of genes had highly divergent expression (defined conservatively here as a more-than-tenfold difference), including many marker genes. OPCs also had conserved expression and 14% highly divergent genes. Two thirds of all genes analysed (9,748) had divergent expression in at least one of 37 homologous types, and many had expression changes restricted to one type or class. Non-neuronal types had the most divergent expression (3,643 genes with more than tenfold difference), supporting increased evolutionary divergence of non-neuronal expression patterns between human and mouse17
Translation: Nearly 20% of genes had very different expression between mouse and human, with up to a greater than ten-fold difference, and at least two-thirds of genes analyzed had different expression between mouse and human. This difference was particularly notable for the gene encoding serotonin receptors:
Serotonin receptors exhibit highly divergent expression between species: four of seven G-protein-coupled receptors and both ionotropic receptor subunits (HTR3A and HTR3B) were in the top 10% most-divergent genes (Fig. 6e). The most-divergent gene families include neurotransmitter receptors, ion channels, extracellular matrix elements and cell-adhesion molecules. Among the top 3% most-divergent genes (Supplementary Table 5), the collagens COL24A1 and COL12A1 and glutamate receptor subunits GRIK1 and GRIN3A were expressed in different cell types between species and were validated to have different laminar distributions in human and mouse (Fig. 6f, g). The cumulative effect of so many differences in the cellular patterning of genes with well-characterized roles in neuronal signalling and connectivity is certain to cause many differences in human cortical circuit function.
Serotonin (5-hydroxytryptamine, or 5-HT), as some readers might know, is a molecule with multiple functions in different parts of the body, where it can function as a hormone, growth factor, or neurotransmitter, among other functions. For instance, it’s stored in platelets, which release it when binding to a clot, where it causes constriction of the blood vessels at high concentrations. (As an interesting aside, at lower concentrations it is a vasodilator.) In addition, serotonin also plays a role in wound healing. In the brain, however, it is primarily a neurotransmitter, where its activity is believed to impact a number of functions, including mood, sexual function, appetite, sleep, memory and learning, temperature regulation, and some social behavior. The most widely known role of serotonin (among the general public, at least) is its role in depression, because of the class of drugs known as selective serotonin reuptake inhibitors (SSRIs), which are used for the treatment of social phobia, anxiety disorders, panic disorders, obsessive-compulsive disorders (OCD), major depression, irritable bowel syndrome (IBS) and eating disorders.
Basically, when two neurons communicate, one will release certain neurotransmitters, like serotonin, into the space between them (the synapse). A neurotransmitter will diffuse from one neuron (the presynaptic neuron) through the synapse to the postsynaptic neuron, where it will bind to its receptor, thus activating a signal in the postsynaptic neuron. After the signal is sent, neurons get rid of the extra neurotransmitter in the synapse through protein transporters that take them up into the cell again after they’ve bound to the receptor and been released. SSRIs inhibit the activity of specific serotonin transporters, which allows the serotonin to linger in the synapse for longer, thus letting it stimulate its receptor longer.
Because of this, in the discussion, the authors note:
Our results demonstrate species divergence of gene expression between homologous cell types, as shown at the single-gene15 and gross-structural level16. These differences are likely to be functionally relevant, as divergent genes are associated with connectivity and signalling, and many cell-type markers show divergent expression. Notably, serotonin receptors are the second-most-divergent gene family, challenging the use of mouse models for many neuropsychiatric disorders that involve serotonin signalling52.
And conclude:
These observations quantitatively frame the debate of whether human cortex is different from that of other mammals10,11, revealing basic transcriptomic similarity of cell types punctuated by differences in proportions and gene expression between species that are likely to influence microcircuit function. Furthermore, these results help to resolve the paradox of failures in the use of mouse for preclinical studies despite conserved structure across mammals52,53, and highlight the need to analyse human brain in addition to model organisms. The magnitude of differences between human and mouse suggests similar profiling of closely related non-human primates is necessary to study many aspects of human brain structure and function. The enhanced resolution afforded by these molecular technologies also shows great promise for accelerating our mechanistic understanding of brain evolution and disease.
Or, as the lead investigator Ed Lein of the Allen Institute for Brain Science in Seattle said in STAT:
“All of the drugs people are trying to develop act on receptors or other molecules,” said neurobiologist Ed Lein of the Allen Institute for Brain Science in Seattle, who led the study, published in the journal Nature. “If the neurotransmitter receptor you’re hoping to target isn’t used in the same cells in humans that it is in mice, then your drug will hit the wrong circuit” and not have the same effect in patients as in lab rodents.
The bottom line is that there are substantial similarities between mouse and human brains, but this study has unveiled substantial difference that were unexpected, in which the neurons in the mouse version of human brain structures use different neurotransmitter receptors. The results of this study will help scientists interpret results in mouse studies better and, hopefully, apply them to human neurophysiology more accurately. In the meantime, whenever you see a study, look for whether it was just done in mice. There’s even a Twitter feed, @justsaysinmice, whose main purpose is to point out hyped studies that were only done in mice:
IN MICEhttps://t.co/QUp4R2oIKh
— justsaysinmice (@justsaysinmice) August 23, 2019
IN MICE https://t.co/3jwoX1m80S
— justsaysinmice (@justsaysinmice) August 13, 2019
https://t.co/gh4ZlkfSno
— justsaysinmice (@justsaysinmice) April 16, 2019
You should follow @justsaysinmice while we’re waiting to see what this new study contributes to making animal models more predictive of human response.

Precision Medicine and Uncertainty
Harriet Hall on November 15, 2016
Tweet
Precision medicine is a development that promises to tailor treatments to the individual patient based on genetic and other molecular and cellular analyses. In his 2015 State of the Union address, President Obama announced the Precision Medicine Initiative. A website for Precision Medicine claims to offer “perfect care for every individual.”
Much of what we read about precision medicine gives the impression that it provides certainty. That’s a false impression. As David Hunter pointed out in a recent article in The New England Journal of Medicine, it may actually increase uncertainty.
He uses breast cancer gene studies as an example. New tools have extracted information about mutations, epigenetic events, and gene-expression alterations in tumors. Products like Oncotype DX, MammaPrint, PAM50 and others contribute information that goes beyond the traditional clinical prognostic factors like tumor size, grade, and nodal status. The new products are based on evidence about the risk of tumor recurrence, but no direct evidence about which specific treatments were more effective.
A study by Cardoso et al., published in the same issue of the NEJM, looked at the possibility of using the 70-gene MammaPrint test to select patients for adjuvant chemotherapy. They stratified patients into high and low clinical risk (C-high, C-low) and high and low genomic risk (G-high, G-low) and randomly assigned them to either get or not get adjuvant chemotherapy. In the C-low, G-high group, there was no difference between those who did and didn’t get chemotherapy; the genomic information was not helpful for patients at low clinical risk. In patients at high clinical risk, the results were equivocal: forgoing chemotherapy on the basis of low genomic risk resulted in a slightly worse outcome, but the difference was small, only 1.5 percentage points, and the study was not powered to determine whether it was statistically significant. This leaves women at high clinical risk with a difficult decision: if their genomic risk is low, they could avoid the toxic effects of treatment, but it might slightly increase their risk of recurrent cancer.
It’s good that studies like this are being done, but the results don’t provide certainty. In this particular study, dividing a continuous variable into two groups (high and low risk) sacrificed precision for interpretability. Different gene-expression products may result in different risk categorizations, and they improve as technology changes and more data becomes available; the MammaPrint data is 14 years old.
Cancer genomics data is problematic for a number of reasons. Different studies find different information. The raw sequences are usually not published, so results remain ambiguous and questionable. There are many variables, like changes in gene copy numbers and tumor sub-clones, that are not taken into consideration. Precision medicine doesn’t really tailor the treatment to the individual; it lumps large numbers of patients together in order to reach reproducible conclusions.
Hunter concludes:
In the future, we are likely to face a potentially bewildering array of probabilities — estimates of disease risk based on inherited germline sequencing and, once a disease is diagnosed, of prognosis and therapeutic options guided by “-omic” and other analyses. Assessing and acting on these probabilities will require approaches to data presentation, risk quantification, and communication of uncertainty for which we are largely ill equipped and that we already struggle with. In most situations, the best advice will be far from obvious and will often rely on a preliminary estimate as the data mature. In parallel to developing the tools for “-omic” analyses, we urgently need to develop methods to help our patients absorb large amounts of complex information that will help them make choices among increasingly numerous options with increasingly numerous trade-offs. These methods should also help our colleagues answer the age-old question, “What would you do, doctor?”
Conclusion: uncertainty remains
In the early days of genome analysis, the promise seemed infinite. We hoped to pinpoint the genes responsible for every illness. We hoped to find certainty and solutions. We are not there yet, and we are beginning to question whether we ever will be. Uncertainty has always been a part of medicine, and it will continue to be. Precision medicine has great potential, but so far it may only be increasing our uncertainty and making decisions more difficult.

WHO Statement on Reporting Clinical Trials
Steven Novella on April 15, 2015
Link


The World Health Organization (WHO) has recently released a new position statement on mandatory reporting of all interventional clinical trials. This is a positive step in the trend towards higher quality and greater transparency in clinical trials.
The underlying ethical concept here is that the public has a right to data that results from experimentation on humans. The researchers do not ethically own that data. They have been granted the privilege of performing research on humans as part of a social contract that includes the timely public reporting of the data.
The WHO statement calls for all registered clinical trial results to be made public in an open-access public forum in a searchable format within 12 months, and publication in the peer-reviewed literature within 24 months. They urge more rapid dissemination, but state that the 12/24 month time frame is the upper limit of when results should be reported.
The WHO statement is part of a larger trend toward greater transparency but also quality in scientific research. It is part of a recognition that we cannot understand how to best practice medicine and allocate resources based upon individual studies. We need to look at the entire scientific literature as a whole. Even when individual studies may be of high quality, there are many potential factors that can distort the scientific literature and therefore misinform doctors, scientists, and regulatory agencies.
Dissemination bias is one such factor. In the past certain pharmaceutical companies have even argued that they own the data from clinical trials they funded. This argument has been soundly rejected. The debate resulted in the formation of clinicaltrials.gov, a site where all human clinical trials must be registered. This registration has allowed for tracking of how clinical trials are reported.
A 2010 study found that trials with positive results were more likely to be published, and are published earlier, than trials with negative results. A 2015 study published in the NEJM found:
Only 13.4% of trials reported summary results within 12 months after trial completion, whereas 38.3% reported results at any time up to September 27, 2013.
Corporate funded studies with FDA oversight were more likely to be reported than those funded by the government or academic institutions – so this is not a problem isolated to the pharmaceutical industry.
A 2014 study in the Annals of Internal Medicine also found many discrepancies between the trials registered in clinicaltrials.gov and what ultimately is reported. These discrepancies included reporting of secondary outcomes, reporting of adverse events, and even reports of deaths.
Conclusion
One of the missions of Science-Based Medicine is to explore the relationship between scientific evidence and the practice and regulation of medicine. We have learned a great deal over the last couple of decades about this relationship, which is complex and evolving.
It is important to understand how to evaluate the quality of individual studies and how the type of information provided relates to the practice of medicine.
Perhaps even more complex, however, is the relationship between the body of scientific evidence and our ultimate conclusions about practice and regulation. We now know that most individual published studies are wrong. You have to look at the overall pattern in the published literature to make a reliable conclusion.
Often we rely upon systematic reviews to assess the overall pattern in the literature. However, such reviews are likely to reflect any biases in the publication and dissemination of research results. There is a remarkable bias toward publishing positive studies – among researchers, funders, and journal editors. There also appears to be a bias toward provocative studies, which are the very ones that are most likely to be false positives. Exact replications, perhaps the most important type of study, are seen as uninteresting. Some journals, as Richard Wiseman discovered, may even have a policy of not publishing boring exact replications.
What is clear is that we need a global effort to improve the technology of tracking and disseminating clinical trial results to minimize any biases. Fortunately we have a relatively new technology, the internet, to help with this effort. There is no longer any excuse of the pages of a print journal being a limited resource. An unlimited amount of data can be published online.
Ideally we would have a situation in which all human clinical trials are registered before they begin, including details about methods, outcome measures, number of subjects, and comparisons (to reduce researcher degrees of freedom). Once registered, trial progress and results will be tracked and reported in a timely fashion. All this data will exist in a public, open-access, and searchable database so that researchers can easily access the information. Negative results will not be swept under the rug. Exact replications will be encouraged and enthusiastically reported.
As a result, systematic reviews addressing a specific medical question will be much more likely to capture the true answer free of systematic biases. Ethics demand nothing less.
Without thorough and transparent reporting, the rights of subjects in clinical trials are violated. They allow themselves to be experimented on for the advancement of science, and if their results are not reported in a timely fashion then that trust is betrayed. Further, any subject in a future clinical trial, that may have been unnecessary if only previous trials were properly reported, are also being treated unethically. Dissemination bias wastes resources, distorts the practice of medicine, and is a disservice to patients, research subjects, and physicians and regulators who rely upon published data.
The WHO statement is a step in the right direction, although it needs to be backed up with more teeth. We now have the technology to make all clinical trial results transparent and accessible. This has the power to dramatically improve the science of medicine. The ethical imperative to do so seems clear.

Open Data
Steven Novella on February 26, 2014
Link
PLOS (the Public Library of Science) is a non-profit open access publisher of science articles. Their goal is to make scientific data accessible to everyone, in the name of transparency and open communication. Now they have taken their approach one step further, announcing their policy that all articles published in a PLOS journal must submit their original data so that anyone can access and analyze it for themselves.
In an effort to increase access to this data, we are now revising our data-sharing policy for all PLOS journals: authors must make all data publicly available, without restriction, immediately upon publication of the article. Beginning March 3rd, 2014, all authors who submit to a PLOS journal will be asked to provide a Data Availability Statement, describing where and how others can access each dataset that underlies the findings. This Data Availability Statement will be published on the first page of each article.
They allow for exceptions—when subject confidentiality is an issue, sensitive information related to endangered species, and when the authors do not own the data. In such cases, however, data must be available upon request, and not controlled by the authors. Otherwise the raw data must be made available.
I think this is a fabulous idea, for many reasons. We frequently write here at SBM about the challenges faced by the various institutions of science to maintain high standards of quality and transparency. Those challenges include publication bias, the literature being flooded with preliminary or low quality research, researchers exploiting degrees of freedom (also referred to as “p-hacking”) without their questionable behavior being apparent in the final published paper, conflicts of interest, the relative lack of replications and lack of desire on the part of editors to publish replications, frequent statistical errors and the occasional deliberate fraud.
There are many ways to erode the quality of scientific research, or to manipulate research to achieve a desired end (rather than discover what is real). In the end, however, I am not nihilistic. Science can and does still move forward, although slowly. We just have to recognize how messy the process is so that we can best sift out the noise and find the reliable evidence.
It does feel as if we are in an era of self-examination and increased efforts to identify and correct the failings of modern scientific research. It further seems as if the transparency and immediate access to data afforded by the internet is largely responsible for this. This is also an era of experimentation where various models are being proposed or tried as potential solutions to the various problems faced by science.
The Open Access movement is one such experiment, and PLOS has been its flagship. In my opinion, the experiment has been a partial success, but has created some of its own problems. It is certainly extremely useful to have immediate access to a full published article when researching a topic. This facilitates post-publication peer review, and the discussion within the community about the research. PLOS has also managed to maintain a reasonably high quality among its journals.
However, open access journals without such high standards have also proliferated. The business model of most open access journals is that they do not have subscriptions (by definition) so they pay for themselves by charging authors a publication fee. This can create a perverse incentive to publish lots of low quality papers in order to garner those fees, and since publication is only online (without the expense of print journals), creating minimalist open access journals with terrible quality control can be profitable. Last year Science published the results of a “sting” operation exposing the poor quality of many open access journals (PLOS, to its credit, did not fall for the sting).
Open access is therefore not a panacea, and comes with its own challenges. It can, however, address the issues of transparency and universal access to facilitate review and discussion. I therefore think it is a great idea for PLOS to go “all in” on this strategy. If you are going for transparency, then make the raw data transparent, not just the final worked-over data.
In fact we have proposed this previously as one strategy to combat the problem of p-hacking. If researchers disclose the process by which they collected data, all the data they collected, and every way it was analyzed, then p-hacking would become more transparent, and this would hopefully discourage the practice. At the very least it would make it easier for other researchers to reanalyze the data to see if the results are genuine or an artifact of creative analysis.
In fact, I think all journals should adopt this policy. Researchers should make available to journal editors all their raw data, so that the journal can make it available either online or on request to other researchers who want to review or reanalyze the data, or just to help them replicate the study.
Conclusion
I think we are in an exciting time in the evolution of the institutions of science. Many problems that have been festering for a long time are being exposed and discussed. This may be unsettling—to learn about all the flaws in the practice of science. But these flaws all have potential solutions, and many of them are not difficult at all, they just need the will to execute.
Journals and their editors are largely the gatekeepers for the official record of scientific research—the published peer-reviewed literature. Therefore many of the solutions to these problems rest with them. Open access is one approach, that I feel will have a long future and play an important role in reforming the institutions of science. Requiring open access to data is a great move that capitalizes on the strength of the open access movement.
Print journal editors, in fact, would be well-advised to follow suit.
In recent years it has become policy for researchers to disclose funding and potential conflicts of interest. Clinical trial registries have also been created so that companies cannot hide research whose results they don’t like. But more reforms still are needed.
Universities should institute more uniform education of researchers so that they are more aware of the problems of p-hacking, to minimize error and bias and to maximize scientific rigor.
Journal editors should publish more negative studies and more replications (including exact replications). Not everything has to end up as a full article in the print version of the journal. Online supplements can provide the space needed to publish whatever is necessary to maximize the unbiased flow of quality scientific information.
Science is often characterized as a self-corrective process. The process of science itself needs to be self-corrective. These fixes do not require massive resources, just reasonable changes in policy. PLOS should be commended for helping lead the way, at least with respect to open access to data.

Researching SBM Online
Steven Novella on July 17, 2013
Link
The internet is a fabulous resource of information. It is one of those technological innovations for which you soon can no longer imagine how you lived without it. I certainly cannot imagine a project like science-based medicine prior to the web.
The web, however, is also a tremendous source of misinformation, opinion, and ideology. Also the volume of information, good and bad, can be overwhelming. We therefore are frequently asked the meta-question of how we conduct our research into specific topics, or how can the average layperson do their own research online.
Efficiently and effectively researching a complex topic is complex. It is a skill that needs to be developed, and it is especially difficult without having detailed knowledge of the specific topic ahead of time. Therefore there is no simple answer to this question, but I can offer some tips.
There are two main resources I use when searching a topic, Google and PubMed. They each have their strengths and weaknesses. For the average user, Google (or whichever general search engine you prefer) is likely going to be your first stop.
Using Google
When I am researching an unfamiliar topic the first thing to pay attention to is the search terms. You may not be using the proper or most precise terms in your search because you are unfamiliar with the jargon. You may need to keep searching with variations of descriptions of the topic in which you are interested. If you know the name of someone or an institution attached to that topic, include that in the search.
When you hit an article that refers to or is related to the topic, pay close attention to the specific terms that are used. They will become your new search terms as you hone in your search. Sometimes it take me several rounds of doing this before I get to the precise terms that get me to the content I want.
The strength of Google is its search capability. The weakness is that the results are going to be contaminated with many low-quality sites from an SBM perspective – meaning sites that are commercial or ideological. Once you find your optimal search terms, adding “skeptic” or “skeptical” to the search may bring you to the critical analysis for which you are looking.
Using PubMed
PubMed is a publicly available database of peer-reviewed journals in the medical field, including basic science and psychology. Overall this is a better resource than Google, but you need a higher level of familiarity with the topic. Often I use Google just to find the proper search terms to use in a PubMed search.
The strengths of PubMed include the fact that you are taken only to peer-reviewed primary scientific sources. The search engine, however, is not as robust or user-friendly as Google, so you really do need to know the technical search terms. You can also, however, search by the name of the publishing author or the journal, if you have that information.
An imprecise search can result in tens of thousands of results. You may have to wade through dozens of articles to find one that is relevant to your topic, but once you hit one relevant article that will likely lead you to the articles you want through better searches. Also, with PubMed, when you click on an article the site will also list other related articles. Look through those, and when you click on them you will get more related articles.
PubMed displays the abstracts of articles only. Abstracts are useful summaries, and may be adequate for your purposes – if all you want is to learn the bottom line. But for any detailed analysis you need to read the full article, as abstracts are often misleading. PubMed offers links to full articles when available, which is increasingly the case, but often the full article is not available online or is behind a paywall (an expensive paywall).
A well-written article will often contain an introduction which provides the authors’ summary of the topic, including an overview of current research. This is often an excellent way to understand the current thinking of researchers about that topic.
The most useful articles on PubMed are often going to be systematic reviews. If you want to go straight to these articles, then just include the word “review” in your search. Reading the three or four most recent systematic reviews of a specific topic or question is the quickest way to get a useful bottom-line summary of the status of the evidence.
Evaluating Resources
Not all resources are equally valid or useful. When it comes to published primary research, the reputation of the journal and the authors are the most important criteria. That is why I will search on the author names (in both Google and PubMed) to find out what other research they have done, or if they are involved in any ideological advocacy.
Luckily journal names often give away their focus and impact. Journals with names like “Nature” and “Science” have a long history and solid reputation. Journals with names like, “The Filipino Journal of Cancer Homeopathy,” likely don’t have the same reputation. But if in doubt, search for other articles in the same journal to see what they are all about.
When evaluating sites on the web, there are a few useful rules of thumb. Group sites are more reliable than sites dedicated to the work of a single individual. University sites tend to be of higher quality than commercial sites. Be suspicious of sites that are trying to sell you something, or that have a clear ideological ax to grind.
Otherwise, the general principles of skeptical analysis apply. Be wary of sites that are promoting conspiracy or fear mongering, or that are making claims that seem too good to be true.
In addition to SBM, another deep resource of medical information on the web is QuackWatch. On the homepage there is a section for “Nonrecommended Sources of Health Advice.” That is an excellent list of sources to avoid, unless of course you are looking for claims to debunk. There is also a list of recommended links.
Putting it all together
Perhaps the most difficult question when researching a specific topic is this – when are you done? When have you found all the relevant information? You may feel as if you have found many valuable resources, but perhaps you are missing that one article that changes everything. Again, there is no simple answer to this. The more time you invest in research, the more confident you can be that you have found all relevant information.
The only way to be sure is to do a systematic search, but that is frankly not feasible for the average person. Even for experts, like the authors of SBM, we rely upon other researchers to do systematic reviews, which can take hundreds of hours.
But, following the tips above, if you are reasonably diligent, you can wrap your head around a narrow topic or specific question with a few hours of online research.
It is always useful, however, to simply ask another person for their opinion, especially if they have more expertise in that area. I will often ask other experts – this is what I have found, am I missing anything?
One of the advantages of blogging is that it serves as a means of crowdsourcing. If, after doing my research, I did miss something, it is likely to pop up in the comments. The only way to be really sure you thoroughly covered a topic is to discuss it with others who have knowledge in the area. You have to engage with the community. That is part of the reason why scientists go to conferences and meetings.
If you are not a blogger, then simply engaging in the comments of a blog like SBM can serve the same purpose. You can also ask questions on forums and message boards. Of course, such online communities are likely to have their own culture and biases. You can easily get sucked into an echo chamber with an extreme point of view. Keeping your sources diverse is important here as well.
In truth the process of researching a topic is never truly done. Even for topics that I feel I have mastered, I am frequently encountering new viewpoints or nuances. Like science itself, it is an endless self-corrective process that requires you to be open to revision.

Reporting Preliminary Findings
Steven Novella on May 30, 2012
Link
While scanning through recent science press releases I came across an interesting study looking at the use of a pharmaceutical grade antioxidant, N-Acetylcysteine (NAC), in the treatment of certain symptoms of autism. This is a small pilot study, but it did have a double-blind placebo controlled design. The press release reports:
During the 12-week trial, NAC treatment decreased irritability scores from 13.1 to 7.2 on the Aberrant Behavior Checklist, a widely used clinical scale for assessing irritability. The change is not as large as that seen in children taking antipsychotics. “But this is still a potentially valuable tool to have before jumping on these big guns,” Hardan said.
But concluded:
“This was a pilot study,” Hardan said. “Final conclusions cannot be made before we do a larger trial.”
I also noticed that two of the authors list significant conflicts of interest – patents on the use of NAC, and one has equity in the company that makes it. It occurred to me that a larger question than the efficacy of NAC for these autism symptoms is this – if this is a pilot study only and we should not base any firm conclusions on the results, then why the press release?
In a broader context, we are currently going through an information revolution. There are many positive aspects to the easy access to information provided by the internet, but it is a disruptive technology in many ways. The news industry is desperately trying to adapt to the collapse of their traditional business model, for example. Relevant to my question above, the scientific community is now exposed to public scrutiny in a dramatic way that is very different from the pre-internet age.
It used to be that studies were published in print scientific journals that would largely be read by experts in the field. Journal articles and presentations at scientific meetings were (and continue to be) an important conversation that each scientific community has with itself, to review new findings and ideas. This conversation happened largely in isolation from public view, because it really is only interesting and relevant to researchers. Most new ideas in science are not going to pan out. The meat grinder of peer-review, replication, and analysis by the scientific community weeds out many hypotheses (which in the field of medicine are potential new treatments).
As new hypotheses survive through the process of peer-review and further research, the valuable ideas will tend to survive while the rest fade away (at least ideally that is what should happen). At some point scientific ideas become established enough that they begin to filter their way into the science section of newspapers, articles in popular scientific magazines, and then finally into documentaries and text books. I won’t pretend this system ever worked perfectly, but at least there was a relationship between the degree to which scientific ideas were validated and their penetration into public awareness. Anyone really interested in the latest research could read primary journals or attend meetings, but preliminary findings were not often presented as news to the public.
The internet has changed this. Now scientific journals often publish their abstracts online, and there are some journals that have full public online access. On the whole I think this is a very good thing. It facilitates research and the exchange of information. I certainly cannot imagine maintaining an information source like Science-Based Medicine, with almost daily articles, without instant online access to published articles.
The scientific community has to realize, however, that as a consequence of this the conversation that scientists used to largely have with each other is now happening more in the public eye. In medicine this means that patients are reading preliminary studies and making health care decisions based upon the findings.
The issue within medicine is similar to the broader issue created by the democratization of information by the internet – the explosion of access (both on the creation and consumption end) increases freedom but decreases quality control (which in medicine we call the standard of care). Journalists argue that a million amateur bloggers are not a replacement for even a single full time, trained, investigative journalist, and they have a point.
I am not arguing for limiting access to information. That genie is out of the bottle, and I think the positive effects of information access far outweigh the negatives. But I do think that the scientific and medical communities need to recognize the new reality they are living in and adapt to it. I have some specific recommendations.
The first is not to send out press releases for preliminary pilot studies. The studies are there for those who are interested to find, but sending out a press release almost guarantees the results will be published as news, and the caveat at the very end about the results being preliminary is likely to be lost on most of the public. Essentially the authors are saying, here are some interesting results, now don’t make any health decisions based upon them. It’s not only unrealistic, it’s irresponsible. Over reporting of preliminary results, most of which are likely to be wrong or misleading, is a disservice to the public. The noise it generates will lead to confusion, and may lead to a decrease in public confidence in the scientific community.
In the case of NAS and autism, the authors acknowledge that NAS as a supplement is already being used by “alternative” practitioners to treat autism. Further, the NAS used in the study is pharmaceutical grade and specially handled to prevent break down of the chemical, and the results (in addition to being preliminary) cannot necessarily be applied to the supplements being sold. So this study is a setup to be misused to promote an existing market for an unproven therapy.
Journals should also consider more clearly labeling preliminary studies they publish, and perhaps even creating a special section in the journal for such studies. The reason for this is that the “peer-reviewed” label gets attached to definitive and preliminary research alike, and often this label is used to promote an idea to the public. If they were published in a section of the journal designated for preliminary findings, that might help reduce confusion over what peer-reviewed actually means. It would at least provide an easy alert to journalists and news outlets – this article comes with a huge warning at the top that the results are preliminary only (not a short caveat at the end).
I also believe that the scientific community needs to be more involved in reporting science news to the public. Press releases are often handled by non-scientists working in the press office of a journal or university, and often they distort or overhype the research. Their goal is not to accurately report results, but to promote their institution and grab headlines. The dwindling science journalism infrastructure means that many news outlets just pass along the press releases, without doing any actual journalism themselves. Often generalist reporters are reporting science news stories they are not equipped to understand. We need to rebuild the science journalism infrastructure by finding ways to support journalists who are trained to report science news, and training scientists to be their own journalists or at least how to deal with the media.
Finally we need to include in our science education curriculum how to skeptically read science news. The public needs to become more scientifically literate, because no matter what steps we take to improve how science news is reported to the public, there will always be a tremendous amount of bad or misleading science reporting (again – the internet genie is out of the bottle).
Conclusion
The scientific community needs to recognize that the world has changed around them. The conversation that scientists used to have mainly with each other is now increasingly happening in full public view. This can ultimately be a powerful boon to science, which needs to be transparent and open. The scientific community, however, has to adapt to this new reality. It has to retool the conversation to account for the public attention, and it has to engage more energetically with the public. Further it should become more directly involved in promoting higher quality science education in our society.

Journal Club
Mark Crislip on July 27, 2012
Link
There is a tradition in medical training called Journal Club. The first rule of Journal Club is you do not talk about Journal Club. In Journal Club, at least in the iterations in which I have participated, one article is selected by an attending, everyone reads it, then the strengths, weakness and applicability are discussed by the group. Usually a top notch, ground breaking article was the focus, one that had high potential clinical impact. But since they were good articles in good journals, there was not a lot to learn about in reguards to critical thinking. While the attending would put the article in context and maybe discuss some rudimentary statistics, there was little that was discussed about the quality of the study. The main take home from every study was to question the applicability of the results to populations that were not old, white males, since it seemed all the ground breaking studies back in the day were a VA Cooperative study of one sort or another.
As I remember it, there was not really a conceptual frame work with which to evaluate studies. Bayes theorem, and its application to clinical medicine was never explicitly discussed outside of testing, where you have to consider the prior plausibility of the patient having a disease before you can decide if the test results is a true positive or not. In Portland, Oregon, the chance that a Lyme serology is a false positive is much greater than a test done in Portland, Maine. Generally speaking in the information overload state that is the practice of medicine, clinical trials are generally taken at face value and tests are considered infallible. Which is a shame, as I wonder how much suboptimal medicine is inflicted on patients by not considering prior plausibility and how accurate a given test is in either ruling in or out a disease. There seems to be a whole industry built around treating patients with no risks for Lyme but have positive tests of doubtful provenance. We never discussed the prior plausibility and its effect on the outcomes of a studied treatment.
I keep coming back to Why Most Published Research Findings Are False by John P. A. Ioannidis as the archetype framework by which to evaluate the truthiness of studies. A problem with applying the Ioannidis paper is that for a lot of big ticket items in medicine the preponderance of data suggests the approximate optimal way to treat common diseases. Brouhaha to follow in the comments as I am sure I will be flooded with counter examples. I speak as a hospital based ID doctor, and most of the time for patients admitted to the hospital we have a rough idea as to what needs to be done diagnostically and therapeutically based on the likely causes of the patients symptoms. There are always fine points deriving from the individual patients comorbidities. At some level, for example, every community acquired pneumonia is the same, requiring a beta-lactam and a macrolide as initial empiric therapy, and every community acquire pneumonia is different, depending on allergies, exposure risks etc. Humans tend to function in relatively narrow operational parameters, although with nearly infinite combinations of those parameters. New papers often have a vast background of similar studies that places any new work in context. I suspect the framework of the Ionanidis paper has more applicability to the new and unproven, to the cutting edge of research.
Reading the medical literature critically as a resident or fellow, there is little need to think about all the ways the literature could be wrong. The assumption is that the good studies in the good journals are mostly right and testing is mostly accurate. It wasn’t until a decade after my training did I start to think critically, or even need to think critically, about the medical literature, and then only as part of my interest in SCAM’s. As a specialist an understanding of the is and outs of the literature comes as part of acquiring the breadth and depth of knowledge of in the areas of my expertise (which is more information than you require). The limitations of a given study are always discussed in the context of the entire literature on a topic. Often it is not that a paper is binary true or false, but often, given the qualifiers of the limitations of a given study, there is a continuum of truthiness for most of the literature, even in a disease as common as pneumonia. I can talk for an hour (a non-addicting substitute for Ambien) on the issues concerning the clinical trials that resulted in the current guidelines for the treatment of pneumonia. Medicine is messy and complex, filled with qualifiers. As to the rest of medicine? I don’t pay that close of attention. I have just have so many neurons I can devote to medicine, so for areas outside of my expertise, I defer to the experts, which is what most of us have to do in a busy day.
Far more can be learned about critical thinking if Journal Club were devoted not to the best of the best, but to the best of the worst, and there is no area of medicine with worse clinical trials that SCAM. One such crossed the LCD this month as I prepared for my Puscast, by way of Medscape*: Meditation, Exercise May Decrease Cold Symptoms said the headline. The authors modestly refer too their study as a “ground-breaking randomized trial of meditation and exercise vs wait-list control among adults aged 50 years and older found significant reductions in ARI illness.”
I love the way ground-breaking trails off into qualifiers. But ground-breaking. This requires more than a skim of the interweb summary, it requires going though the original. I canna pass up ground-breaking, now can I?
There is always the unreliable gut check to start. I read the title and think, ‘that can’t be true’ or ‘that’s interesting’, and then read the paper and, ‘meh, the gut check was wrong again’ or ‘cool, I’ll try to remember this.’
I tend to have that horrible, western, reductionist metaphor when thinking about human physiology and pathophysiology: we are meat machines. Americans are often poorly maintained meat machines with suboptimal diets and insufficient exercise. Although my gut reaction was regular exercise should decrease the riskfor infectious diseases, I did not know the data. The same week the meditation article was released, in my literature search I found Physical Activity and Influenza-Coded Outpatient Visits, a Population-Based Cohort Study , which suggested
Moderate to high amounts of physical activity may be associated with reduced risk of influenza for individuals < 65 years.” It was not a comparative clinical trial but a data mining evaluation comparing “physical activity levels through survey responses, and influenza- coded physician office and emergency department visits through physician billing claims. We used logistic regression to estimate the risk of influenza-coded outpatient visits during influenza seasons. The cohort comprised 114,364 survey respondents who contributed 357,466 person-influenza seasons of observation. Compared to inactive individuals, moderately active (OR 0.83; 95% CI 0.74–0.94) and active (OR 0.87; 95% CI 0.77–0.98) individuals were less likely to experience an influenza-coded visit. Stratifying by age, the protective effect of physical activity remained significant for individuals ,65 years (active OR 0.86; 95% CI 0.75–0.98, moderately active: OR 0.85; 95% CI 0.74–0.97) but not for individuals $65 years.
A search of the Pubmeds reveals a smattering of studies that demonstrate both no exercise and excessive exercise increase the risk of upper respiratory infections with moderate exercise being in the Goldilocks zone for benefit. The data also shows that immune function, however they chose to measure it, is better with moderate exercise. The meat machine runs better when active. In epidemiological studies there is always the chance that the perceived cause of benefit, in this case exercise, is only a marker for other reasons for the effect: those who exercise have other factors that decrease risk. Health and disease are never as simple as they appear at first glance. But I have few doubts concerning the multitudinous benefits of regular, moderate, exercise.
My first reaction to meditation was bah, humbug. The theory: stress makes one susceptible to infection, meditation can decrease stress, therefore meditation, by receiving stress, will decrease infection risk. The authors say
There is some evidence that enhancing general physical and mental health may reduce ARI burden.In a series of observational and viral inoculation studies, perceived stress, negative emotion, and lack of social support predicted not only self-reported illness, but also such biomarkers as viral shedding and inflammatory cytokine activity. Evidence suggests that mindfulness meditation can reduce experienced stress and negative emotions.
A bit of a stretch perhaps, but interesting if it pans out. Stress is always a tricky one in the practice of medicine. Just as every patient seems to perceive they are uniquely susceptible to bad luck, in my practice it is an unusual infection: “if something bad is going to happen, it is always me.” I have never had a patient say, “that’s odd Doc, I am always so lucky, it is weird how this bit of misfortune affected me”, patients often perceive themselves under inordinate stress. Still, the data does suggest that stress and personality type may increase the risk of infection, so it is plausible if one could decrease stress with meditation, you could decrease susceptibility to infection.
I am inclined to think the premise behind the study is reasonable based on prior research, and it would be a trial that if well done would be further evidence, as if people follow evidence, of the benefits of exercise. In their trial not only was exercise of benefit in preventing acute respiratory illness (ARI), but mindfulness meditation was even better than exercise at preventing acute respiratory problems.
We observed substantive reductions in ARI illness among those randomized to exercise training, and even greater benefits among those receiving mindfulness meditation training.
Unfortunately the trial has perhaps every known flaw one can make in a clinical study and it renders the results useless, as much as I would like them to be true.
The lead author, Dr. Bruce Barret, has been supported by NCAAM in the past and I suspect may have a different approach to applying clinical trials to medicine than I do. His response to a negative trial of ehinacea for the treatment of colds study was
Adults who have found echinacea to be beneficial should not discontinue use based on the results of this trial, as there are no proven effective treatments and no side effects were seen.
The antithesis of an EBM/SBM approach to medicine, but I do not know if it is representative of Dr Barret’s general approach to medicine. I would have said based on the data, echinacea is crap, doesn’t work, has no reason to work, so quit using it and don’t waste your money. I mention this only because bias of all kinds can color the approach to a trial and its interpretation and this trial is open to huge amounts of inadvertent bias.
Always the most difficult issue in a study: bias.
Corollary 5: The greater the financial and other interests and prejudices in a scientific field, the less likely the research findings are to be true. Conflicts of interest and prejudice may increase bias, u. Conflicts of interest are very common in biomedical research, and typically they are inadequately and sparsely reported. Prejudice may not necessarily have financial roots. Scientists in a given field may be prejudiced purely because of their belief in a scientific theory or commitment to their own findings. Many otherwise seemingly independent, university-based studies may be conducted for no other reason than to give physicians and researchers qualifications for promotion or tenure. Such nonfinancial conflicts may also lead to distorted reported results and interpretations.
It is why double blinding is so important in clinical trials, as the endless ability of patients and researchers to fool themselves and each other as to benefit is enormous. Without careful double blinding patient and researcher are no more than Clever Hans.
Problems with the trial: small numbers of patients. While they report outcomes on 149 patients, which is almost respectable, the protocol was actually run twice, once in the fall with 91 patients and once in the spring with 58 patients, and the combined data reported. Different viral seasons, but it appears to be two trials reported as one and data is not reported from each individual study. It is more a meta-analysis of two small, flawed studies rather than one larger flawed study.
There are multiple comparisons for both primary and secondary outcomes. When there are small numbers of patients and multiple comparisons, more often than not anything ‘significant’ is more likely due to random scatter than real effect.
But the fatal flaw was the lack of blinding. Of course the patients and the researchers knew who was receiving what intervention. It would be difficult to invent placebo exercise or meditation. Patients were called twice a week and if they reported ARI symptoms they received a laboratory evaluation within three days of onset. Not only were patients aware of their assignment, but the study relied on self-reporting to determine if they were starting an ARI. If would have been more impressive if every patient had a laboratory evaluation and nasal swab for pathogens twice a week, regardless of symptoms.
One clear result of the NEJM article was that patients who receive a placebo intervention perceive themselves as better even when they are not. So relying on the patients perception of becoming ill while in the meditation or exercise wing of the group is instantly suspect. Given the potential for poisoning the well, the lack of blinding renders the results useless. Given that the perceived effect of acupuncture depends mostly on the patients belief that acupuncture will have an effect, one wonders how much expectation lead to the improved results in the meditation group. I wonder what results would occur is patients enrolled in NCAAM funded SCAM studies were members of the JREF or CSI. So much opportunity for the clinical trial equivalent of the Stockholm effect, trying to please your researcher.
It was a preliminary study, so flawed as to hardly be ground-breaking, more in the maybe interesting if it were actually done in a way where the data was meaningful. It is the kind of study I would like to be validated. If you use the title as a Google search term, it appears the article is being used more to justify the meditation aspects than the exercise and as a validation of alternative and complementary medicine in general.
At the end of the day this article at best elicits a meh, so filled with flaws as to almost be a waste of the ink and paper it was printed on. But that is the way of clinical research. Really crappy preliminary trials whose results are either flat out wrong or markedly overstated will, one hopes, be superseded by better designed trials where the decline effect will kick in. Better trials, I predict, well demonstrate the benefits of exercise for decreasing the odds of infection and the dramatic benefits of mindfulness meditation will drift towards the insignificant. And the results of this particularly flawed study will persist longer than any subsequent trial that suggests otherwise: “Adults who have found meditation to be beneficial to prevent colds should not discontinue use based on the results of this trial, as there are no proven effective treatments and no side effects were seen.”
The JREF will owe me a cool million.
* remember, I am a paid Medscape blogger and writer.

Standards in Behavioral Science
Steven Novella on July 2, 2025
Link
There is an excellent review in The Independent (by Helen Coffey) about the recent cultural shift within behavioral science. It reflects the exact same issues that we address here with medical science – identifying and eliminating shoddy scientific practices. It’s worth going over, providing examples from the medical side.
Perverse Incentives
The goal of science should be to discover the truth, regardless of what it is. This is especially important for an applied science like medicine – we want our interventions to be safe, effective, cost effective, efficient, and minimally invasive and disruptive. To achieve this we need to know what actually works – we need the best, most reliable science possible.
But there are other incentives that get in the way. Researchers want positive results that confirm their biases. Positive results are also more likely to be published and advance one’s career. It helps if the results are surprising and interesting, what Coffey calls “sexy”. Journal editors also like sexy results, because that increases the visibility, prestige, and impact factor of their journal. The press adores sexy results because they are very media friendly.
I would extend this idea of perverse incentives to include play to pay journals that just want to publish lots of stuff, regardless of quality. And of course there are ideologues who want to promote their particular world view. This gets entangled with the financial and career goals above as well. The result is that, if you, say, are an acupuncturist, you want to publish studies that show acupuncture works. You cite other studies that show acupuncture works. You can create in the literature an acupuncture fiction that has nothing to do with reality and is built on all the shoddy research that Coffey discusses.
It goes even further than Coffey realizes, because the sexy but shoddy research is not just a one-off. It can be part of a campaign promoting an entire false idea, even a false cultural institution. Things like homeopathy, acupuncture, Reiki, megavitamins, antioxidants, and chiropractic take on a life of their own. You get journals, institutions, and even entire professions dedicated to nonsense.
But at the core of all of it is the bad study. So let’s review Coffey’s points and even add some.
Fraudulent Studies
I don’t need to say much about fraudulent studies except that, unfortunately, they do exist. We can collectively do a better job of policing against fraud, detecting it, and weeding it out quickly, preferably before it ever gets published. This is mostly on journals and their editors, who need better fraud detection practices.
But perhaps the most important thing to realize about fraudulent scientific research is that this is not the main problem. It is a destructive and horrible thing, but relatively small compared to good-faith but sloppy research.
P-Hacking
P-hacking refers to superficially fine but ultimately questionable research practices that essentially amount to statistical cheating. The p-value is a rough statistic that is used to see if a study is even interesting – are the results likely to be a statistical fluke or the result of a real phenomenon. But the p-value is widely misunderstood and overused. Even worse, it has become too much of a focus of research, and has led to (sometimes inadvertent) hacking to get significant results.
Essentially these are methods that distort the statistical results by giving more throws of the dice (often without disclosing this fact). So, you can collect data until the results become significant, or make multiple comparisons, or look at multiple outcomes.
There are several basic fixes for p-hacking. One is to simply educate researchers about methods of p-hacking to make sure they don’t accidentally do it. But also editors can specific try to detect p-hacking and demand data from submissions that would help them do it. But there are two more definitive fixes. One is pre-registration of study methods. You cannot p-hack if you determine all research methods before collecting data. The other is replication, which follows the original methods to see if the same results occur.
Fragile Studies
Even if a study is honest and does not engage in p-hacking, the results may still not be reliable or generalizable because they are “fragile” (which you can think of as the opposite of being robust). A fragile study, for example, looks at a study population which is not representative for some reason. Coffey uses the example, common in behavioral psychology, of only using college students. But any study with narrow inclusion and exclusion criteria can also be fragile. Perhaps the results are only positive in certain cultures or subcultures (such as the fact that acupuncture studies are way more likely to be positive if conducted in an Asian country).
Another source of fragility is a small sample size. Fifty subjects in each arm of a study is generally considered to be a good minimum for a statistically robust study, and less than that should be immediately suspect. This depends on the outcome, however. More objective outcomes, like death, can get away with smaller sample sizes, while subjective outcomes like pain perception require even larger studies.
I would also consider another sign of fragility that only one laboratory or researcher can seem to generate positive results. Until a result reliably replicates, it is suspect.
Also, all the little details of studies that we often discuss in our specific reviews can be filed under fragility. A study may have a large drop-out rate, or not be properly blinded, or use dubious outcome measures, or a host of other weaknesses in the protocol.
Salami Slicing
Coffey uses the term “salami slicing” to refer to what is also called the sharpshooters fallacy, or more generically the problem of hypothesizing after you look at data. The sharpshooters fallacy refers to determining what a positive outcome is after seeing the result you already have, like shooting at the side of a barn and then drawing the target around your bullet hole, declaring who got a bullseye.
Ideally a research study would start with a clear hypothesis, and a specific method for gathering data that will test that hypothesis in a way that makes a-priori sense. Salami slicing is the practice of collecting a lot of data, then slicing and dicing up the data in different ways until you find some correlation that is significant, then backfilling some justification for why that’s the case.
Results of this approach often lack what we call face validity – they don’t seem to make sense on their face. But the results can be statistically significant, or at least appear significant if you don’t know and correct for the fact that multiple comparisons (slicing of the data) were done.
Importantly, the results can often be surprising and sexy – sure, because they also happen to be bullshit.
The Future
Coffey ends her piece on a positive note, saying that exposing all these dubious methods is having an impact, improving the overall rigor of the scientific literature. I agree that this is happening, although I would argue in needs to happen at least an order of magnitude more than we are currently seeing.
But there are also some counter-trends. At the same time we are trying to shore up the rigor of biomedical science, there are forces trying to weaken those standards, or at least carve out exceptions for their prefers beliefs. They have political allies, and lots of money.
Also, the media appears to be working against us. To make this point, just look at the ads below Coffey’s article. They represent the exact thing she is discussing. Social media seems to be designed, even more than mainstream media, to favor sexy results. There is now a cottage industry of influencers, self-help gurus, self-appointed pseudoexperts, contrarians, snake oil peddlers, and ideologues leveraging social media to spread the absolute worst shoddy science.
The fact that, behind the scenes, we are incrementally improving the rigor of our science is great. But it is overwhelmed by the deluge of misinformation and shoddy science out there. We need to tackle that realm as well. This requires increased standards not just for published studies, but for press releases, public communication, scientific journals, and academia. And we need to dramatically increase the amount and quality of our public science communication.
We also need to dramatically improve the quality of our regulations, which are steadily being ratcheted in the direction of snake oil. Don’t expect any improvement in the next four years, but this is an endless struggle we must keep up.



CONCEPTUAL FRAMEWORKS

What is Science?
Kimball Atwood on February 3, 2012
Link
Consider these statements:
…there is an evidence base for biofield therapies. (citing the Cochrane Review of Touch Therapies)
The larger issue is what constitutes “pseudoscience” and what information is worthy of dissemination to the public. Should the data from our well conducted, rigorous, randomized controlled trial [of ‘biofield healing’] be dismissed because the mechanisms are unknown or because some scientists do not believe in the specific therapy?…Premature rejection of findings from rigorous randomized controlled trials are as big a threat to science as the continuation of falsehoods based on belief. Thus, as clinicians and scientists, our highest duty to patients should be to investigate promising solutions with high benefit/risk ratios, not to act as gatekeepers of information based on personal opinion.
–Jain et al, quoted here
…
Touch therapies may have a modest effect in pain relief. More studies on HT and Reiki in relieving pain are needed. More studies including children are also required to evaluate the effect of touch on children.
Touch Therapies are so-called as it is believed that the practitioners have touched the clients’ energy ﬁeld.
It is believed this effect occurs by exerting energy to restore, energize, and balance the energy ﬁeld disturbances using hands-on or hands-off techniques (Eden 1993). The underlying concept is that sickness and disease arise from imbalances in the vital energy ﬁeld. However, the existence of the energy ﬁeld of the human body has not been proven scientiﬁcally and thus the effect of such therapies, which are believed to exert an effect on one’s energy ﬁeld, is controversial and lies in doubt.
—Cochrane Review of Touch Therapies, quoted here
…
Science is advanced by an open mind that seeks knowledge, while acknowledging its current limits. Science does not make assertions about what cannot be true, simply because evidence that it is true has not yet been generated. Science does not mistake absence of evidence for evidence of absence. Science itself is fluid.
—David Katz
…
When people became interested in alternative medicines, they asked me to help out at Harvard Medical School. I realized that in order to survive there, one had to become a scientist. So I became a scientist.
—Ted Kaptchuk, quoted here.
…
…It seems that the decision concerning acceptance of evidence (either in medicine or religion) ultimately reflects the beliefs of the person that exist before all arguments and observation.
—Ted Kaptchuk, quoted here.
Together they betray a misunderstanding of science that is common not only to “CAM” apologists, but to many academic medical researchers. Let me explain.
Science: You keep using that word. I do not think it means what you think it means.
In common use the word “science” has several meanings, which may or may not be clear in context. Two that readily come to mind are 1) the growing body of knowledge about nature, accumulated over the several hundred years during which a distinctive, rational method of inquiry, or at least parts of it, have been employed; 2) that method of inquiry, also known as the “scientific method,” characterized by the collective tools of science—observation, generation of hypotheses, controlled and repeated testing of hypotheses, the use of mathematics for generating hypotheses, for aiding in complex measurements, for statistical inference, and so on.
The most important misunderstanding about science that we see in the quotations above is a conflation of the two meanings, or in some cases a complete disregard for the first. Thus, because some clinical trials have suggested “a modest effect on pain relief” for (non)Touch Therapies, there must be “an evidence base for biofield therapies,” and to argue otherwise is “as big a threat to science as the continuation of falsehoods based on belief.” Moreover, since “absence of evidence” is not the same as “evidence of absence,” and since “science does not make assertions about what cannot be true,” the responsible scientist must judge the “energy field of the human body” to be a real possibility, even if it “has not (yet) been proven scientiﬁcally.”
Such notions are nonsense: of course science makes assertions about what cannot be true. Nor will clinical research ever overturn those assertions, which are based on far more rigorous and voluminous experimental data than can be generated in the messy, incorrigible realm of clinical trials. It is the first meaning of science, as defined above, that is the more pertinent for “CAM” research. The second meaning is barely applicable, for reasons that I’ve discussed previously and that I may revisit here, if I have the time.
“What have we been doing for the past 300 years?”
The late physicist, Milton Rothman, wrote three small books that are useful for a discussion such as this. One of those books, A Physicist’s Guide to Skepticism, has an entire section titled “Laws of Permission and Laws of Denial.” The chapter on “Laws of Denial” begins as follows:
It is fashionable in some circles to insist that “nothing is impossible,” as though to admit the impossibility of some cherished goal is to “give up trying,” to have a closed mind, to be a spoilsport, a pessimist. This cliché is most prevalent in inspirational rhetoric connected with therapeutic, educational, or sporting activities. Nevertheless, one of the basic functions of science is to determine what actions are impossible in this real world. Choosing between the possible and the impossible is a task carried out by means of the laws of denial, which tie us firmly to reality even as imaginations soar unfettered through the universe.
Another fashionable cliché is that “all scientific theories are provisional,” as though physics knows nothing with a certainty, and that anything we think we know is likely to be found false in the future…If all scientific knowledge is tentative, what have we been doing for the past 300 years? How can I be so sure that the computer upon which I am typing will print out the words that I am putting into it?
A more accurate assessment of the situation is to recognize that one of the fundamental tasks of science is to critically examine all knowledge and to separate from the tentative ideas and false notions of the past facts that are so well established that to think them subject to change is to invite wishful thinking and foolishness.
Laws of denial, as explained by Rothman, are the laws of conservation of energy, momentum, angular momentum, and of electric charge; the principle of Lorentz invariance, “from which the conclusions of special relativity follow: no object, energy or information can travel faster than the speed of light”; the principle of causality, by which it is “impossible for an effect to appear earlier in time than its cause”; and the first and second laws of thermodynamics. There are other statements that can be made with a degree of certainty much higher than is necessary to preclude their being overturned by clinical research, even if they are less certain than the laws of denial. For example, since all known interactions can be explained by the 4 forces of the standard model, and since only two of those forces—gravity and electromagnetic force—explain all actions other than those at the subatomic level, there is no reason to invoke fanciful forces (the vital force, ‘biofields’) that have never been detected and that add nothing to our understanding of natural phenomena.
Rothman distinguishes between “ideological skepticism” and “pragmatic skepticism”:
Ideological skepticism is disbelief based on deep-seated psychological factors…It includes disbelief in conservation of energy and other laws of denial because you can’t stand authority figures telling you what you cannot do. [It] encourages you to think that we can’t know anything for a certainty, and that, as a result, anything is possible.
Pragmatic skepticism is disbelief in phenomena that contradict laws of nature that have been thoroughly verified by experiment and observation. It is based on a well-founded understanding of those natural laws, and of their uses and limitations.
It should be clear that pragmatic skepticism—the kind that makes real scientists and SBM authors skeptical of “biofields,” homeopathy, psychokinesis, and all the rest—is based on neither “personal opinion” nor “the beliefs of the person that exist before all arguments and observation.”
Rothman asks, “how do the laws of physics give us the power to make any statements about biology or psychology, about evolution or human behavior?”
The answer lies in the laws of denial. While we are unable to make good predictions about what things will do using the laws of permission, we can make very precise predictions about what they cannot do.
Rothman then offers a few negative predictions of his own, with explanations based on the laws of denial. In summary:
     1.  I will never be able to jump as high as the moon, at least not without mechanical aid.
     2.  I will never suddenly burst into flames.
     3.  I will never suddenly levitate and rise up off the floor, no matter how hard I will it…
     4.  No one will ever build a flying vehicle that is capable of hovering high in the air while supported by nothing but magnetic fields.
     5.  No one will build an antigravity machine…
     6.  No one will ever build a time-travel machine.
     7.  No one will ever make a killing on the stock market by foreseeing the future.
     8.  Nobody will ever send a message through space that does not diminish in intensity as it travels away from the sender.
     9.  No one will ever send or receive any kind of message that travels faster than the speed of light.
     10.  No one will ever influence the position or motion of any kind of physical object from a distance just by thinking about it.
     11.  No one will ever demonstrate that astrology really works.
Number 10, a reference to psychokinesis, pertains to a large category of “CAM” claims, including ‘biofields,’ Therapeutic Touch, and ‘distant healing.’
When you think of science, please think of it first as a way—the only accurate way that we have—to understand nature. If, instead, you think first of P-values or confidence intervals or randomization or blinding or allocation concealment, you’re misled. Some of those may be important tools for some kinds of research, but they do not constitute science any more than rulers or scales or graph paper or chromatography constitute science, and those whose expertise is limited to such tools may be smart and useful for some scientific pursuits, but they are not scientists.
“A Subtle Change in the Balance of Medical Authority”
Nearly 4 years ago we saw these quotations from a real biomedical scientist, Steven Goodman, who also happened to have learned Bayesian inference:
An important problem exists in the interpretation of modern medical research data: Biological understanding and previous research play little formal role in the interpretation of quantitative results. This phenomenon is manifest in the discussion sections of research articles and ultimately can affect the reliability of conclusions. The standard statistical approach has created this situation by promoting the illusion that conclusions can be produced with certain “error rates,” without consideration of information from outside the experiment. This statistical approach, the key components of which are P values and hypothesis tests, is widely perceived as a mathematically coherent approach to inference. There is little appreciation in the medical community that the methodology is an amalgam of incompatible elements, whose utility for scientific inference has been the subject of intense debate among statisticians for almost 70 years…
This method thus facilitated a subtle change in the balance of medical authority from those with knowledge of the biological basis of medicine toward those with knowledge of quantitative methods, or toward the quantitative results alone, as though the numbers somehow spoke for themselves.
That appears to explain why “More studies on HT and Reiki in relieving pain are needed,” and why they will continue to be ‘needed,’ ad nauseam. And now, thanks to commenter phayes, I’ll go back to my new, favorite treatise on how to use probability theory to make sense of incomplete information. Hint: it isn’t what the frequentists who now claim “medical authority” use.
…
The Prior Probability, Bayesian vs. Frequentist Inference, and EBM Series:
1. Homeopathy and Evidence-Based Medicine: Back to the Future Part V
2. Prior Probability: The Dirty Little Secret of “Evidence-Based Alternative Medicine”
3. Prior Probability: the Dirty Little Secret of “Evidence-Based Alternative Medicine”—Continued
4. Prior Probability: the Dirty Little Secret of “Evidence-Based Alternative Medicine”—Continued Again
5. Yes, Jacqueline: EBM ought to be Synonymous with SBM
6. The 2nd Yale Research Symposium on Complementary and Integrative Medicine. Part II
7. H. Pylori, Plausibility, and Greek Tragedy: the Quirky Case of Dr. John Lykoudis
8. Evidence-Based Medicine, Human Studies Ethics, and the ‘Gonzalez Regimen’: a Disappointing Editorial in the Journal of Clinical Oncology Part 1
9. Evidence-Based Medicine, Human Studies Ethics, and the ‘Gonzalez Regimen’: a Disappointing Editorial in the Journal of Clinical Oncology Part 2
10. Of SBM and EBM Redux. Part I: Does EBM Undervalue Basic Science and Overvalue RCTs?
11. Of SBM and EBM Redux. Part II: Is it a Good Idea to test Highly Implausible Health Claims?
12. Of SBM and EBM Redux. Part III: Parapsychology is the Role Model for “CAM” Research
13. Of SBM and EBM Redux. Part IV: More Cochrane and a little Bayes
14. Of SBM and EBM Redux. Part IV, Continued: More Cochrane and a little Bayes
15. Cochrane is Starting to ‘Get’ SBM!
16. What is Science? 

How Unbekoming! The central delusion of MAHA
The antivax movement has a central conspiracy theory, but there is also a central delusion behind MAHA, revealed by a blogger with the ‘nym Unbekoming.
David Gorski on May 25, 2026
Link

A long time ago, in—seemingly—a galaxy far, far away (or at least a time that now seems very far, far away indeed), I coined a term, the central conspiracy of the antivax movement. In brief, this conspiracy theory posited that vaccines are harmful, causing autism, autoimmune disorders, cancer, etc., and that “They”—the medical profession, the CDC, the FDA, state governments, and the federal government—know they are causing massive harm but are covering up evidence of that harm. The motivations for the “coverup” vary according to the weaver of the conspiracy theory and generally involve one of or a combination of motivations involving greed, ideology, and control, but the central conspiracy theory has long remained the same. Vaccines are evil; “They” know vaccines are evil; and “They” are covering up and/or actively suppressing the evidence uncovered by the brave maverick doctors and rag-tag vaccine “resistance” conclusively demonstrating that vaccines are evil. Now that longtime rabid antivax activist Robert F. Kennedy, Jr. is Secretary of Health and Human Services and thus in charge of all non-military federal medical, public health, and biomedical research programs, from Medicare/Medicaid, to the Centers for Disease Control and Prevention, to the Food and Drug Administration, to the Public Health Service, to the National Institutes of Health, we are seeing the “unbekoming”—I’ll explain shortly—ideas behind that conspiracy theory become policy under the banner of the “make America healthy again” (MAHA) movement, to the detriment of public health in the US and, more broadly, the world.
This central conspiracy theory of the antivax movement goes beyond just the antivax movement, of course. Its more general version, that there are “natural” cures to diseases that “They” don’t want you to know about, dates back generations and has long been a key feature of the various movements that have existed around alternative medicine, “health freedom,” New Age woo, and the like. Unsurprisingly, this sort of conspiratorial thinking now underlies much of RFK Jr.’s MAHA movement, but there is also an associated central delusion that now underlies the MAHA movement, after having long been a core feature of alternative medicine. Nowhere do I recall having seen it laid out more clearly than in a recent post on a Substack that I recently discovered. The Substack is entitled Lies Are Unbekoming. (I really hate that cutesy stylistically spelling of “Unbecoming.”) The post is entitled You Are Born Defective™: An Essay. (Yeah, whoever the pseudonymous writer of this Substack, who goes by the ‘nym “Unbekoming,” is, they’re clearly very full of themselves.) In this “essay,” Unbekoming claims that the main problem with medicine is that it views human beings as having been “born defective” and in need of correcting.
Unbekoming is clever, though. Here’s how the post is introduced:
This essay operates in two registers. When the medical system’s claims are being examined, the system’s own language is used — vitamin K deficiency, immune system, mutation, cholesterol threshold. This is the establishment being prosecuted in its own words, using its own evidence. When the position the essay actually holds is being stated — that the body is self-healing, self-regulating, and intelligent — the language shifts. The body’s lymphatic and cleansing network is not an “immune system” of warriors and defenders. The MTHFR polymorphism is not a “mutation.” Cholesterol is not a “danger” the body inflicts on itself. The dual register is the structure of the argument: the establishment’s framework is allowed to speak so its contradictions can be heard, and then the terrain framework names what is actually being observed.
There is a huge “tell” in this mass of pseudoprofound, pseudointellectual blather that reveals clearly where Unbekoming is really—if you’ll excuse the term—coming from, and that’s the use of the term “terrain framework.” I’ll discuss that in bit more detail later, but for the moment, “terrain theory” is the 19th century idea championed primarily by one of Louis Pasteur’s competitors, Antoine Béchamp, that it’s not the viruses, bacteria, etc. that are causing disease, but rather that disease is caused by the “terrain” being somehow “damaged” or otherwise hospitable to the outside invaders or causes of disease. It’s an idea that is not entirely untrue. After all, the elderly, the debilitated, those with immune systems that are not functioning properly, whether due to inherent disease or to chemotherapy, are more prone to infection. However, alternative medicine believers—and now MAHA stans—frequently vastly overplay this relationship to argue that the microbes don’t matter much, if at all.
Unbekoming starts with birth and the vision of a human baby that is, well, basically perfect, or nearly so.
Unbekoming: Birth, vitamin K, and vaccines
Unbekoming’s central straw man argument is that conventional science-based medicine views humans as “born defective,” and from that flows the second straw man that medicine must “correct” all these “defects.” To illustrate this, Unbekoming cites the example of the routinely recommended neonatal dose of vitamin K, portraying it in the most “unnatural” and “chemical” language possible, which I will quote in full, even though it’s long, so that you can see clearly what is being done, namely how Unbekoming claims to be “allowing” the “establishment’s framework” to “speak,” which is not actually what is happening here:
The baby is minutes old. The cord has been cut — too soon, but that is another story. The mother has not yet finished holding the child against her chest. A nurse approaches with a syringe.
The syringe contains 1mg of synthetic phytonadione suspended in polysorbate 80, propylene glycol, and a preservative system that has changed several times over the decades the injection has been administered.¹ The dose is roughly 20,000 times what the newborn would receive in a day of feeding on colostrum.² The baby is healthy by every visible measure — pink, breathing, alert, gripping a finger when one is offered.
The injection goes into the thigh muscle. The baby cries. The mother is told this is routine. She is rarely told what the injection is correcting.
What the injection is correcting, the medical system maintains, is a defect. The newborn, in the first days of life, produces clotting factors at levels lower than adult reference ranges. This lower level is called “physiologic deficiency”³ — a term whose internal contradiction passes without examination. The body is producing exactly what its own developmental program calls for, and this is described as a deficiency.
Within the first hour of life, the medical system has identified something wrong with the baby and acted to correct it. This is the first transaction. The baby has been told, on its first day, that the body it arrived with is inadequate.
Note the framing here. The infant is “healthy,” and the evil (and/or deluded) medical system is insisting on ripping it from its mother’s breast to inject it with unnecessary artificial chemicals to correct a what is portrayed as a nonexistent “defect.” (Never mind that babies aren’t ripped from their mother’s breast to be injected with vitamin K. The recommendation is simply that the newborn infant receive an intramuscular dose of vitamin K before leaving the hospital.) The neonatal vitamin K shot is then represented as the first of many “transactions” in which the medical profession portrays the body as inherently “defective” and offers a “correction,” or, as Unbekoming puts it:
There is a paradigm operating beneath modern medicine that is rarely stated as a paradigm, because to state it would be to expose it. The paradigm is that the human body, as it arrives, is defective. It produces too little of some substances and too much of others. Its immune capacities are insufficient. Its genetic code carries errors. Its measurements drift, with age, into pathological ranges. Each of these defects is the basis for a product or procedure that corrects it.
This passage literally made me chuckle out loud, given that this verbiage, minus the portrayal of ripping a baby from its mother’s arms to inject it with toxic chemicals, describes the very foundation of a branch of alternative medicine much favored by MAHA and practiced by MAHA bigwigs like former Surgeon General nominee Dr. Casey Means and Dr. Mark Hyman, namely functional medicine (FM). FM basically does what Unbekoming spends the Substack post accusing modern medicine of doing, but on massive doses of steroids. Basically, FM involves massive overtesting for diseases and conditions in the asymptomatic or “worried well,” with some FM panels involving hundreds of blood markers. Abnormalities in these panels of tests, often repeated at least once a year, are then “corrected” by selling supplements targeting every abnormality, whether clinically relevant to anything or not. Functional medicine, as I like to say, is based on reams of useless tests in one hand and a huge invoice in the other. Unsurprisingly, Unbekoming loves functional medicine to death.
How’s that for allowing Unbekoming’s own contradictions to be revealed by simply quoting what is written?
Vitamin K is an essential factor for one arm of the clotting system in mammals. As we have explained many times over the history of this blog when discussing vitamin K fear mongering by antivaxxers, the reason for the vitamin K injection is that neonates can be relatively deficient in vitamin K because it doesn’t easily cross the placenta and is relatively deficient in breast milk, which can leave infants relatively deficient until they start eating other foods at around six months of age. As Steve Novella noted six years ago, one dose of intramuscular vitamin K at birth “significantly reduces that risk (relative risk in that study was 0.19, with a number needed to treat of 74),” immediately increasing “vitamin K levels, with most of the vitamin being stored in the liver where it is slowly released over the next six months.” He also notes that the risks are minimal. In an earlier post, Steve noted that recent studies looking at infants who developed VKDB showed that most or all of them did not receive vitamin K prophylaxis. For a more in-depth discussion of why a routine neonatal dose of vitamin K is safe and effective, Dr. Clay Jones has you covered. Yes, these posts are several years old, but, seriously, the arguments made by people like Unbekoming against the routine vitamin K dose for neonates haven’t changed, nor has the evidence supporting this practice, other than more data showing it to be safe and effective.
Unbekoming does at least admit that the condition that the neonatal vitamin K injection is designed to prevent, namely Vitamin K Deficiency Bleeding (VKDB), is real; so there’s that. It’s even acknowledged that there are three subtypes of VKDB:
    • Early onset VKDB (within 24 hours): Rare; usually associated with maternal medications (e.g., anticonvulsants, warfarin).
    • “Classic VKDB” (days 2–7): The most common form without vitamin K prophylaxis; GI bleeding, umbilical bleeding, or post-circumcision bleeding.
    • Late onset VKDB (weeks 2–12): The most dangerous form — intracranial hemorrhage occurs in 30–60% of cases and carries significant mortality and neurological morbidity. Exclusively breastfed infants are most at risk.
It’s even acknowledged that vitamin K prophylaxis eliminates the risk of late onset VKDB, which is cited as having an incidence of between one in 14,000 and one in 25,000 infants. (Hint: There are roughly 3.6 million births per year in the US, which, if you accept Unbekoming’s numbers, would translate into potentially 144-257 cases of late onset VKDB, a catastrophic condition, per year. There used to be a time when we thought that preventing hundreds of dead or severely neurologically injured infants was a good thing, but apparently not to MAHA’s way of thinking anymore. (RFK Jr. and HHS have backed away from routinely recommending neonatal vitamin K injections, and more parents are refusing the shots.)
But it’s all so “unnatural.” Unbekoming goes on and on about the excipients used for intramuscular vitamin K injections:
The injection is not vitamin K alone. It contains polysorbate 80, a surfactant shown in animal studies to alter blood-brain barrier permeability¹⁰; propylene glycol, which the FDA itself has warned about in neonates¹¹; and benzyl alcohol, which has been associated with “gasping syndrome” and infant deaths.¹² The product carries a black box warning — the FDA’s most serious — that reads “severe reactions, including fatalities, have occurred during and immediately after intravenous and intramuscular injection,” noting that “some patients have exhibited these severe reactions on first receiving phytonadione.”¹³ A 1992 study from the British Medical Journal reported an association between intramuscular vitamin K and childhood cancer.¹⁴ Subsequent studies disputed the finding. The dispute is unresolved. The injection continues.
First of all, phytomenadione is simply vitamin K1. Second, the dispute is not “unresolved.” Since the study mentioned, epidemiological studies have consistently failed to find an association between the birth dose of vitamin K and increased risk of any cancer. The purported “risk” at this time is purely hypothetical and therefore far outweighed by the benefits of the neonatal vitamin K injection.
As for polysorbate-80, unmentioned is that the animal studies cited used large doses of polysorbate-80, far above what a human infant receives in a vitamin K injection. (Indeed, the quantities of excipients like propylene glycol and polysorbate-80 in current vitamin K injectable formulations in the US are minuscule, making this attack just another variation of the “toxins” gambit beloved of antivaxxers.) Most were studies of polysorbate-80 in the diet, and most recent studies of polysorbate-80 and the blood-brain barrier that I could find involved using polysorbate-80 in lipid nanoparticles designed specifically to target the blood-brain barrier. Indeed, the study cited by Unbekoming to support the fear mongering about polysorbate-80 is a 1985 study in which levels of chemotherapeutic agents were measured in the brains of mice when dosed with and without polysorbate-80 included in the mice’s diet. Surely, if this were a major concern, they could have come up with a more relevant (and recent) study. At least, if I were trying to make the same argument, I’d find something less than four decades old that directly addressed the claim. Ditto benzyl alcohol, for which the reference cited comes from 1982 and looked at doses 100 times higher than what is in vitamin K shots, and even if parents are concerned about the presence of this chemical in the shots there is a preservative-free version of intramuscular vitamin K. Not that any of this stops Unbekoming from citing the litigation-driven antivax pseudo-“study” the Control Group as evidence for the dangers of vitamin K injection.
[NOTE ADDED AFTER PUBLICATION: Ironically, after I finished this post, I noticed that antivax physician Suzanne Humphries showed up on the website of the antivax org founded by RFK Jr. parroting basically the same claims about benzyl alcohol, the now disproven elevated risk of childhood cancer from the shot, and even the claim that the lower vitamin K levels in neonates is “by design” for a purpose. Yep. Unbekoming is parroting old antivax nonsense about vitamin K shots.]
Finally, Unbekoming asks what they must think to be a reasonable question: Why not just give the vitamin K orally in divided doses, as though it were pure ideology—”the hospital workflow is built around injection”—and grift that led the American Academy of Pediatrics to recommend one intramuscular dose of vitamin K1 in 1961 and reaffirm the recommendation in 2021. In reality, a single intramuscular dose provides protection for months. In contrast, oral regimens require multiple doses and have a higher failure rate, particularly for late-onset VKBD. If you listen to Unbekoming, though:
The newborn is not deficient. The newborn is producing vitamin K at the level the newborn’s developmental program calls for, in a body whose clotting system is calibrated to that level. The “deficiency” is defined against an adult reference range that has no biological relevance to a three-day-old infant. A genuine clinical bleeding risk exists in a small subset of infants with underlying pathology, and that risk is real and serious. The response is to inject every newborn, mask the signal that would have identified the at-risk subset, and accept the toxic burden of the carrier substances in exchange for protection against a condition most infants would never have developed.
The reason that the deficiency is defined the way it is is because at the lower level of vitamin K in neonates, the clotting system doesn’t work as well, and there actually is a risk of catastrophic bleeding. I also wonder what “signal” that would identify the “at-risk subset” is being “masked” by a universal neonatal vitamin K dose. What would that “signal” consist of? What action would Unbekoming recommend when an “at-risk” infant is identified? This is basically the same sort of MAHA appeal to “individualized” care that is, in reality, an excuse for undermining public health interventions.
Unsurprisingly, Unbekoming moves on to make the same sort of “appeal to perfect nature” to argue against the childhood vaccine schedule, portrayed this way:
The premise of the schedule is that the child’s lymphatic and cleansing network cannot adequately respond to childhood exposures without injected antigens, adjuvants, and excipients. Stated baldly, this is implausible. Children have been having childhoods for as long as there have been children. The lymphatic system, the fascial network, the mucosal surfaces, the liver, the kidneys — these are not new structures awaiting pharmaceutical activation. They are the body’s cleansing and adaptive architecture, refined across the entire arc of human existence.
“Implausible.” You keep using that word. I do not think it means what you think it means. Yes, children have been having childhoods as long as there have been children, and children have been dying of infectious diseases as long as there have been children. In the 19th century, huge numbers of children died of what are now vaccine-preventable infectious diseases, to the point where it was common for families to lose one, two, three, or even more of their children before they grew up. It’s a straw man to say that the child’s immune system “can’t” adequately respond to childhood exposures. The problem is that the immune system needs training. If it hasn’t seen an antigen before, it will not do as well dealing with that antigen as it will if it has seen it before. Vaccines do nothing more than prime the immune system by letting it see antigens of potentially dangerous disease-causing microbes in a safe manner that doesn’t cause disease, so that it “remembers” them when it encounters them for real.
The rest of Unbekoming’s section on vaccines consists of a lot of well-worn, common, and easily debunked antivax talking points, such as the “toxins” gambit, the “no saline placebo” gambit, the false claim that unvaccinated children are healthier than vaccinated children, and a truly brain-dead analogy:
The injection volumes themselves are worth pausing on. A two-month-old infant weighing roughly 5kg may receive, in a single visit, vaccines totalling around 2.5ml of injected material. Scaled to the body mass of a 70kg adult, the equivalent volume would be approximately 35ml — over two tablespoons of injected pharmaceutical product in a single visit.²⁹ No adult medical protocol approaches this density. The infant receives it in a body still building the structures the substances are delivered into.
Here’s a hint: Vaccines are not like standard pharmaceuticals. The doses do not scale linearly with weight; that’s not how the immune system works. Moreover, 35 ml is not a large fluid volume for an adult, just as 2.5 ml is not a large fluid volume for a two-month-old infant. Whether Unbekoming doesn’t know these—i.e., is just ignorant—or does know these things but knows that their audience doesn’t—i.e., is lying—I don’t know. Take your pick.
Either way, Unbekoming’s fallacy is the same for both vitamin K and vaccines:
Both pillars rest on the same proposition: the body is defective at baseline. The proposition is not stated this way in medical literature. It is implicit in the structure of every intervention. Each procedure, each prescription, each diagnostic threshold rests on the same foundation — that the body, as it arrives and as it ages, fails to meet a standard the medical system has the authority to set and the products to address.
There is another paradigm. It is older than germ theory. It survived the institutional capture of the early twentieth century only at the margins, in the work of Béchamp, Bernard, Tilden, Shelton, Price, and their inheritors. The paradigm holds that the body is a self-healing, self-regulating, intelligent organism. It maintains its own equilibrium. It does not make mistakes. It does not attack itself. It responds to insult — toxic, nutritional, electromagnetic, emotional — with intelligent processes aimed at restoring balance. Symptoms are not malfunctions; they are the body’s response to conditions that warrant a response. Disease arises from terrain conditions, not from baseline defects.
There it is, as expected, the appeal to terrain theory. While it’s true that terrain theory was a legitimate competitor to Pasteur’s germ theory of disease in the mid-19th century, the bottom line was that it lost the race. It was a hypothesis that seemed reasonable at the time, given what was known about medicine, but ultimately failed to be validated as science progressed, just like so many other hypotheses and theories over the history of medicine and science. Scientific evidence and experiments increasingly supported Pasteur and did not support Béchamp. Yes, it’s true that “terrain” contributes to one’s susceptibility to disease, but unfortunately people like Unbekoming seem to make like Bill Maher and claim that if one’s “terrain” is maintained through diet, exercise, and other “virtuous” behavior with respect to health practices, then they are not susceptible to disease and don’t need vaccines. Bob Costas had the perfect retort to such arguments: “Oh, come on, Superman!“
As for the body being “self-healing,” “self-regulating,” and “intelligent,” there is some truth to that, but, like quack-lovers everywhere, Unbekoming massively overstates the case. The body can “self-heal” and “self-regulate,” but there are definite limits, and the system doesn’t always work correctly (e.g., in the case of auto-immune diseases and cancer). As for “intelligence,” the brain produces intelligence, but the body’s homeostatic systems are more akin to algorithms than intelligence.
Naturally, all of this “reasoning” (if you can call it that) leads to the conspiracy theory that science-based medicine promotes vitamin K and vaccines for infants and children because it is creating “diseases” or “deficiencies” of the oh-so-perfec human body in order to profit, which, again, is truly rich given how much quacks and, more recently, MAHA have promoted supplements for nonexistent diseases:
This paradigm has no product line. It generates no patents. It cannot be billed. A body that needs only to be supported — through clean food, clean water, low toxic burden, low stress, sunlight, movement, sleep — produces no revenue. The medical system as currently structured cannot operate on this paradigm. It requires the other.
Oh, you sweet summer child! (Either that, you know damned well that a body that “needs only to be supported” can be monetized by selling it products that allegedly help it with “clean food, clean water, low toxic burden, low stress, sunlight, movement, sleep.”)
Into the adult world
I’m not going to spend as much time on the two adult examples that Unbekoming uses to try to make the same “argument” (such as it is), specifically methylenetetrahydrofolate reductase (MTHFR) mutations (referred to as “defective” genome) and statin treatment for hypercholesterolemia (referred to as “the defective adult”). Once again, I chuckled out loud when I saw that Unbekoming was going to “go there” when it comes to MTHFR mutations, given that testing for such mutations has become such a cottage industry among quacks, MAHA-aligned and otherwise, that eight years ago genetics experts at the Cleveland Clinic called out all the dubious tests for MTHFR mutations being marketed. Most of the time genetic testing for MTHFR mutations.
Just to remind readers, folic acid is not metabolically active itself. It must first be converted to dihydrofolate (DHF) and then to tetrahydrofolate (THF) through enzymatic reduction, a reaction that is catalyzed by the enzyme DHF reductase (DHFR). After that, THF can be converted to the biologically active l-methylfolate by the enzyme methylenetetrahydrofolate reductase (MTHFR). This key conversion is necessary to provide l-methylfolate for the one-carbon transfer reactions needed for purine and pyrimidine synthesis during DNA and RNA assembly, for DNA methylation, and to regulate homocysteine metabolism. Thus, MTHFR is the critical enzyme for almost all biologic processes that involve the metabolism of folate and methionine.
Here’s a diagram illustrating its role:
Formation of l-methylfolate from folic acid. CH2, methylene; CH3, methyl group; DHF, dihydrofolate; DHFR, dihydrofolate reductase, F, folic acid; H, hydrogen; SHMT, serine hydroxy-methyl transferase; THF, tetrahydrofolate.
To be honest, I almost think that Unbekoming might be calling out the quacks here, because, seriously, most science-based medical professionals aren’t going to be routinely ordering MTHFR mutation testing for patients, certainly not for patients complaining of “fatigue,” but a whole lot of MAHA-aligned quacks, particularly functional medicine quacks and naturopath quacks, will. I wanted to assume that, but in reality this whole example serves mainly as an excuse to attack folic acid supplementation as “artificial” and “toxic,” a “chemical” that wasn’t synthesized until 1943. Meanwhile, two common variants, C677T and A1298C, are portrayed as totally “natural” and harmless because approximately 40% of the population carry them.
In reality, this is all a pretext for nothing more than attacking the addition of folic acid to foods—referred to, tellingly, as being “added through industrial fortification”—and the recommendation that people who are pregnant or who plan on becoming pregnant supplement with folic acid. I’m not going to go into gory detail on this topic, because, unsurprisingly, it’s a lot more nuanced than Unbekoming portrays it:
A patient is told they carry the mutation. They are told it predisposes them to cardiovascular disease, pregnancy complications, neural tube defects, depression, autoimmune conditions, and detoxification impairment. They are told they must supplement with methylated folate — 5-MTHF — for life, because they cannot adequately process folic acid.³¹
Yes, the most dreaded pregnancy-related consequence of folate deficiency is neural tube defects in the developing fetus, which is why folic acid supplementation has been routinely recommended during pregnancy. Interestingly, the reference cited doesn’t actually say this. It’s all about folate metabolism and how supplementation with folic acid prevents all the bad things that can happen with folate deficiency, such as birth defects and other adverse pregnancy and health outcomes. What does it say about L-methyfolate?
Not much:
Although folic acid supplementation to supraphysiologic levels has demonstrated many of the benefits to pregnant women and fetuses noted above, the potential risk of high-dose folate supplementation must also be considered. First, folate supplementation can mask vitamin B12 deficiency (pernicious anemia) and care must be taken with susceptible individuals to avoid missing this diagnosis. Also, concerns have been raised about the potentially untoward effects of unmetabolized synthetic folic acid with regard to cancer, depression, and cognitive impairment. 52 With all these concerns, early data suggest supplementation with l-methylfolate rather than folic acid may mitigate these risks.53
See what I mean? (Note that L-methylfolate and 5-MTHF are the same substance.) All this review says is that using 5-MTHF instead of folic acid might mitigate potential effects of unmetabolized folic acid and that “newer research in this area has focused on supplementation with l-methylfolate rather than folic acid as a means of preventing folate-related pathology,” not that the standard recommendation is that if you have an MTHFR variant that interferes with folate metabolism you will definitely need to take 5-MTHF for the rest of your life. Of course, this review is from 2011—Unbekoming seems incapable of citing newer literature—and so I looked up some newer literature. As of November 2024, the American College of Medical Genetics, for instance, recommends against routine screening for MTHFR polymorphisms/mutations and that “women of childbearing age should take the standard dose of folic acid supplementation to reduce the risk of neural tube defects as per the general population guidelines.” As for the general population, it recommends that “MTHFR positive individuals may decide to take vitamin B and folic acid supplements. Although safe (toxicity is rare), evidence is lacking on whether such supplements reduce the risks associated with hyperhomocysteinemia or MTHFR genotype status.”
That sure doesn’t sound like how Unbekoming is portraying modern medicine and how it deals with MTHFR mutations. What Unbekoming describes sounds a lot more like how functional medicine quacks and naturopaths deal with MTHFR variants. Be that as it may, Unbekoming concludes here:
The genome is not defective. The diet is. The defect-claim has been moved from the food supply, where it belongs, to the patient’s DNA, where it absolves the food supply and creates a permanent customer for the supplement industry. The genome, which existed long before the synthetic compound it is being declared inadequate to process, is now held responsible for the inadequacy of the synthetic compound.
The funny thing is, the medical consensus with respect to MTHFR variants seems to be similar: that these are not “defects” and that, unless you’re trying to become pregnant, you probably don’t need to do anything other than eat a good diet with green, leafy vegetables. If you are trying to become pregnant, supplement with folic acid to prevent neural tube defects and other defects in your baby.
The next example, “the defective adult,” is basically your standard anti-statin nonsense. Pretty much everything written by Unbekoming is nonsense of the sort I once wrote extensively about when COVID quack Pierre Kory started to embrace statin denial; so I won’t go into much detail regarding the claims, although I do love a claim like this:
In 1984, the cholesterol threshold for treatment in adults was 240 mg/dL.³⁴ In 1993, it dropped to 200 mg/dL. In 2001, the threshold for high-risk patients dropped to 100 mg/dL. In 2004, it dropped again to 70 mg/dL for very high-risk patients. The 2004 update was authored by a panel of nine members, eight of whom had financial ties to statin manufacturers.³⁵ Each threshold reduction expanded the eligible patient population by tens of millions. Statin prescriptions in the United States exceeded 200 million annually by the 2010s, generating revenue in the tens of billions.³⁶
Yes, and blood pressure thresholds for the diagnosis of hypertension have dropped through the decades as well. It’s what happens as more epidemiological and clinical evidence accumulates and guideline organizations realize that there is a benefit to intervening at lower values, which Unbekoming misrepresents by discussing a hypothetical patient with elevated cholesterol who was prescribed a statin:
The doctor does not mention that the threshold he now fails was lowered by a committee in which eight of nine members held financial ties to the manufacturers of the drug he has just been prescribed.
Here we go with the conspiracy theories again.
This leads Unbekoming to repeat what is, after the previous three examples, a now-familiar refrain adapted for cholesterol and statins:
Cholesterol is not dangerous. It is required. The body manufactures it in the quantities the body requires. When measurements are taken and compared to thresholds, the thresholds are being moved to expand the patient population, not to track a biological change. The “high cholesterol” of a healthy adult is the cholesterol level a healthy adult produces. Treating it with a drug that interrupts a biosynthetic pathway producing multiple essential compounds is not correction of a defect. It is induction of a deficiency.
Again, it never occurs to Unbekoming that the body might actually malfunction, that it might make too much of a substance and that too much of that substance might actually cause health problems. Given this framework, I have to wonder why the article didn’t mention type 2 diabetes, with the argument that impaired glucose metabolism is normal for the person with type 2 diabetes and that metformin and GLP-1-based drugs are all a big scam to induce a glucose deficiency.
What ties them all together? According to Unbekoming:
The four pillars share an architecture. A measurement is taken or a category is named, a threshold is set or a defect is claimed, a product is offered. The threshold is positioned as biology; the product is positioned as correction.
In each case, the body is held to a standard the body did not set. The newborn is measured against an adult reference range with no biological relevance. The child is measured against an immune theory that requires injected substances to function. The adult is measured against a synthetic compound invented in 1943. The aging adult is measured against a threshold lowered by a committee paid by the manufacturer.
In each case, the actual cause of the symptoms or risks the intervention claims to address is foreclosed. The newborn’s bleeding risk, where it exists, is from underlying pathology that the universal injection masks. The child’s chronic disease load is from the toxic burden of the schedule itself, alongside diet and environment. The adult’s MTHFR symptoms are from a diet built around fortified processed food. The adult’s cardiovascular risk is from inflammation driven by diet, stress, toxic exposure, and metabolic dysregulation — not from cholesterol levels the body itself maintains.
In each case, the patient becomes a customer for life. The newborn injection initiates the relationship. The schedule extends it. The MTHFR diagnosis cements it. The statin completes it. From the first hour to the last, the body has been a revenue stream, and the defect-claim has been the mechanism by which the revenue stream was constructed.
There it is, the central delusion of MAHA, that the body is this all-knowing, (nearly) all-powerful self-healing machine and that the only thing that prevents it from healing itself of virtually anything are the “toxic load” of chemicals and pharmaceuticals that the medical profession nefariously hooks you on by declaring your body to be less than perfect, full of “deficiencies” that must be corrected for the rest of your life. Given that viewpoint, it should not surprise anyone that Unbekoming is a virus denier, basically claiming that viruses don’t really exist in posts such as What Is Ebola? and What Is Hepatitis C? Amusingly, Unbekoming argues that Ebola doesn’t really exist, that it is really arsenic poisoning, much as antivaxxers did for the polio virus in the past, and “poisoning” due to other industrial toxins related to mining. (Everything old is new again.) Meanwhile, hepatitis C isn’t due to a virus, but just “toxins” that are “poisoning” the liver. (Everything old is new again.)
As Unbekoming shows us, the central delusion of MAHA now (and a lot of alternative medicine in the past that has now glommed onto MAHA) is that the body is basically perfectly functional, even when it’s not. The lower level of vitamin K in the neonatal bloodstream is exactly as nature intended, and the neonatal birth dose is an abomination that “masks” those who are truly “at risk” for VKDB at the cost of exposing all babies to supposedly horrific “toxins.” Your immune system is already exactly as nature intended it to be, and vaccines only interfere with that perfection, again at the cost of exposing the body to horrific “toxins.” Your genome is also exactly as nature intended it to be, and food alone is enough to prevent neural tube defects if you happen to have an MTHFR variant that leaves you deficient in folate. If you have elevated cholesterol, it’s just as nature intended and not at all harmful, even though medical research tells us that there is increasing risk of cardiovascular disease with elevated levels of LDL and cholesterol.
This delusion leads to a single answer to nearly all medical problems related to chronic disease. A medically virtuous diet and lifestyle will fix everything and prevent all diseases—I only exaggerate slightly—and you don’t need pharmaceuticals or vaccines. Would that this were true, but unfortunately we live in the real world. Our bodies are not perfect. Far from it. Most of us will need medical care at some point in our lives, often at many points. The central delusion of MAHA is to deny this essential reality.


Defining what a “physician” is
It takes more than a stethoscope and a white coat to be a physician. A generous dose of reality and a lot of training are also required.
David Gorski on November 14, 2011
Link
The very concepts of “complementary and alternative medicine” (CAM) and “integrative medicine” (IM), the former of which “complements” science-based medicine with quackery and the latter of which “integrates” pseudoscience-based with science-based medicine are all about slapping a veneer of scientific legitimacy onto something that has failed to achieve such legitimacy through actual basic, translational, and clinical science. The reason I start out by saying this is to emphasize that CAM/IM is all about using language to persuade that pseudoscience is actually science-based. It’s far more about marketing than accurately communicating concepts. In CAM, everything is “holistic,” and doctors “care for the whole patient,” while “Western medicine” is “reductionistic” and “allopathic.” At the very heart of this language is a false dichotomy: That you must either embrace pseudoscience or that you somehow can’t provide care as compassionate and caring as what the quacks supposedly provide, nor are you able to provide for the emotional needs of your patients. There are two false dichotomies, actually, in that there is also the not-so-subtle implication in CAM that you can’t be truly “holistic” without—you guessed it—embracing the pseudoscience that is at the heart of many CAM/IM modalities.
This use and abuse of language for propagandistic purposes in CAM/IM is not limited to just these examples. In fact, the misuse of language infuses the whole enterprise of CAM/IM to the point that its adherents, not content with being mere “practitioners,” are trying to claim the very title of “physician” for themselves. I learned this from John Weeks, the main force behind the Integrator Blog, a blog dedicated to issues of CAM and IM. He’s the one who first let me know about Andrew Weil’s attempt to put together a board certification in IM. In particular, his reporting on the reaction of CAM/IM practitioners, both physicians and non-physicians, to this initiative by Andrew Weil was most illuminating to me. What was most telling was how further propagandistic use of the language focused on “dominance” by MDs, which in this case struck me as actually being closer to the truth than the usual CAM-speak is. In any case, Dr. Weil’s initiative does indeed appear to be more about taking control of CAM for physicians, his high-minded language about “establishing standards” notwithstanding.
This time around, Weeks has provided me with an education about how alternative/CAM/integrative practitioners now covet the title of “physician”. In the process, he also uses and abuses language in the same way that Andrew Weil and CAM/IM advocates do. This time around, it’s all about co-opting the title of “physician” for non-physician CAM practitioners. It’s bad enough to me when actual physicians are seduced by the pseudoscience of CAM, but this effort appears to be an intentional strategy designed to confuse the public by proclaiming as physicians practitioners who lack the essential skills to be a physician, such as acupuncturists, chiropractors, homeopaths, and naturopaths.
Weeks begins with the story of a non-physician (i.e., non-MD, non-DO) practitioner of CAM who distinguished chiropractors, naturopaths, and acupuncturists from physicians. To me, of course, that’s an utterly appropriate distinction (it’s not for nothing, after all, that I frequently refer to the naturopath’s ND title as meaning “not a doctor”), but to Weeks it’s “stepping into a militarized zone”:
A doctor level, non-MD academic friend from a conventional medical school who has expertise in complementary and alternative medicine inadvertently stepped into a militarized zone the other day. On a conference call with a multidisciplinary group of practitioners, he wished to make a distinction. He first offered a generalization about the roles of “chiropractors, naturopaths and acupuncturists.” Then he distinguished these from what he called “physicians.”
All of us knew that this studied academic was referring to licensed medical doctors, a.k.a. the kings (and queens) of the hill in U.S. medicine. At the same time, many of us were acutely aware by his comment that, while common and even professional usage of the term “physician” may connote MDs, legal realities are shifting. In fact, trends suggest that one endpoint of the movement toward integrative medicine is that “physician” will increasingly umbrella a rainbow of disciplines.
It turns out that there is a conscious effort under way by promoters of CAM/IM to appropriate the title of “physician” for themselves or, at least, to dilute it down so that it can mean either MDs or any of a number of other types of health care practitioners. Surprisingly, they’ve apparently even had some legislative and regulatory success. Weeks reports that the United States Department of Labor recently published something called O Net Online which replaces the Department’s Dictionary of Occupational Titles and that in that online dictionary there is reference to “naturopathic physicians” and “chiropractic physicians.” Worse, if you look under the sample of reported job titles for chiropractors, a synonym for chiropractor is apparently just “physician.” I also couldn’t help but notice that the descriptions of these jobs are pretty funny in a morbid sort of way, for example, listing one of the functions of a naturopath as administering “treatments or therapies, such as homeopathy, hydrotherapy, Oriental or Ayurvedic medicine, electrotherapy and diathermy, using physical agents including air, heat, cold, water, sound, or ultraviolet light to catalyze the body to heal itself.” Yes, it’s depressingly accurate, but it’s funny in that there is no recognition that homeopathy is quackery, as is much of what naturopaths do.
Even acupuncturists are apparently getting in on the act, trying to claim the title of “acupuncture physician” and even proposing an abbreviation of AP to describe themselves. In Florida, for instance, acupuncturists are legally entitled to work under this title, although “not through accredited, doctoral-level educational means.” Of course not. Acupuncturists are not physicians, nor should they ever be referred to as physicians unless they also have an MD or a DO degree. (And in that case I’d seriously question whether a physicians who has become an acupuncturist and practices acupuncture deserves the title of “physician” any more.) Somehow, apparently acupuncturists managed to get the legislature to give them the right to refer to themselves as “physicians,” and boy do they take advantage of it! Indeed, during my visit to Florida last week to present Grand Rounds at the Florida State University College of Medicine, Jann Bellamy and I discussed this very issue.
But it’s not just acupuncturists. Naturopaths have been getting in on the action, too. Indeed, naturopaths in 15 of the 16 states that license them are allowed to call themselves “physicians.” (Apparently California is the only holdout willing to put its foot down.) But in the case of naturopaths, it goes far beyond that. Besides wanting licensure in as many states as possible, naturopaths are pursuing in these licensing laws the right to be able to practice as primary care providers (even though they have zero expertise in primary care) and even as highly specialized practitioners, such as naturopathic oncologists, the latter of whom tack the abbreviation FABNO, which stands for ““Fellow of the American Board of Naturopathic Oncology.” Meanwhile, a couple of miles away from where I practice, across the border, Canadian naturopaths keep trying to get the right to be able to prescribe real medications, either as an admission that their woo is mostly useless or because they want to be considered “primary care physicians.” Maybe a little of both. Be that as it may, naturopaths in other states, such as New Hampshire, are clamoring for laws mandating that health insurance plans pay for their services:
Republican members of the House Commerce Committee bucked their leadership Thursday and voted 15-5 to endorse a health insurance mandate requiring insurance companies to cover people who use naturopathic practitioners as primary care physicians.
If the bill is approved, the insurance companies of patients of some 60 practices throughout the state will have to reimburse the visits.
“Patients have been begging us to push for this legislation,” said Dr. Jaclyn Chasse, president of the New Hampshire Association of Naturopathic Doctors. “We couldn’t be happier.”
I can’t help but notice how the New Hampshire organization is called the “New Hampshire Association of Naturopathic Doctors,” much as the national organization is called the American Association of Naturopathic Physicians (AANP), not to be confused with the American Academy of Nurse Practitioners. I also can’t help but note that a typical nurse practitioner is far more able to provide primary care services than any naturopath. Yet naturopaths keep trying to wrap themselves in the mantle of the term “physician.”
Weeks is shockingly (to me, at least) honest about the reason for this push by promoters of unscientific health care modalities to be able to call themselves “physicians,” using as an example chiropractors:
Success in claiming the physician title, linked to privilege, status and particularly third party payment – some insurers will only cover certain services if provided by a “physician” – figured heavily in an October 2, 2009 mailing to members from the American Chiropractic Association (ACA). The ACA credited its hard work for insuring that the title stayed in the language defining the Federal Employee Benefit Plan. Blue Cross Blue Shield, which manages the plan, attempted to demote chiropractors to a status as “other health care providers.” (6)
As I’ve said before, it’s all about the status. Chiropractors, acupuncturists, naturopaths, and other specialties of woo crave the respectability that comes with being a physician. They also have delusions of grandeur in thinking that they can do what physicians do, namely provide primary care services, even though none of them have adequate training in the administration of drugs or the science-based screening for cancer, heart disease, and other common diseases to function as primary care doctors, among other functions of primary care. Nor can they deal with common diseases, such as diabetes, hypertension, coronary artery disease, or others, at least not using science- or evidence-based guidelines. Unfortunately, this delusion of grandeur leads them to want to blur the line between real physicians and practitioners of pseudoscientific medicine.
Not surprisingly, Weeks is very unhappy at even quite reasonable attempts by physicians to protect the title of “physician” by opposing the licensing of naturopaths, as if it were a bad thing to prevent the state from putting its imprimatur on specialties that are not based in science and evidence. He also confirms something that I’ve always suspected about the near-obsessive use of the word “allopathic” to describe conventional MDs and DOs:
Then again, the AMA leaders in keeping “physician” out of reach of other professions are probably not as comfortable with the additional 2008-2009 language: “MDs are also known as allopathic physicians.” The modifier “allopathic” is a hole in the dike through which may flow naturopathic physicians, chiropractic physicians, perhaps more acupuncture physicians, and the first through, osteopathic physicians.
In other words, modifiers are the way to go, as far as letting non-science-based practitioners claim the title of “physician.” If physicians who practice science-based medicine are described not just as “physicians” but “allopathic physicians,” then they are lowered and other woo peddlers claiming the title of physician are elevated to approximate equality. Then you have various flavors of “physicians,” including “allopathic,” “naturopathic,” “chiropractic,” and “acupuncture” physicians. I do give Weeks credit, though, for mastery of deceptive language on par with the best propagandists. After all, if osteopathic physicians are now equivalent to MDs, even though 50 years ago they were viewed with extreme suspicion, then why not naturopathic or chiropractic physicians?
I’ll tell you why not.
The reason that DOs are now generally considered on par with MDs is because the way that osteopathic physicians (DOs) practice medicine, at least in the U.S., is usually indistinguishable from the way that MDs practice medicine. In essence, DOs became the equivalent of MDs by changing their education and practice to become like those of MDs. They now do the same residencies, are subject to the same standards and board certifications, and are licensed in the same way as MDs. In most osteopathic medical schools, osteopathy has been deemphasized to the point where it’s now, more than anything else, a historical vestige that continues to be taught more because of tradition than because of any continued enthusiasm for osteopathy. Indeed, very few DOs actually ever use it or practice it anymore once they graduate. Most, in fact, are rather embarrassed by these reminders of the chiropractor-like beliefs that used to be central to osteopathy. We can only hope that naturopaths go that way, but it’s highly unlikely, given the sheer extent of woo practiced by naturopaths, who are not only not embarrassed by their woo but downright proud of it. Remember, homeopathy is still considered an integral part of naturopathy, so much so that it is a major part of the curriculum of every school of naturopathy whose curriculum I’ve ever examined and is also a requirement on the naturopathic board examinations. (Don’t get me started on the very concept of a naturopathic board examination.)
Be that as it may, if you want to get an idea of how far Weeks and presumably his non-science-based holistic cohorts are willing to go to redefine what a “physician” is, look to the last part of Weeks’ post, where he actually invokes Webster (as in the dictionary) to convince his readers that physician should mean what he wants it to mean. He lists various definitions of “physician” that he’s found in different versions of Webster, picking the ones he likes, the ones that are seemingly most inclusive. In other words, he exercises one of the lamest arguments known to humankind. When you’re reduced to arguing based on dictionary definitions, you’ve sunk about as low as you can go.
My amusement at such lame arguments notwithstanding, I can’t resist pointing out Weeks’ one last envious broadside at physicians:
The physician term is power. Professions so denominated stand at the top of the hill, waving about highfalutin reasons why their recognition as such is in the public good. The sub-text includes the know motivator that the title lands one on the road to the bank.
I must conclude by saying again that I’m pleasantly surprised at just how honest Weeks is about the motivation for trying to redefine the term “physician” to include quacks. It has nothing to do with improving patient care, nor does it have anything to do with quacks being able to provide primary care that is equivalent to that provided by real physicians. Rather, it’s all about power, money, and prestige, and the quacks want as much of all three as physicians have. Never mind that they haven’t actually earned such prestige and power through proving the value of what they do to society and through validating their contributions to health care though science. They want it all anyway. Unfortunately, they appear to be succeeding in co-opting the term “physician,” diluting its value to the point of meaninglessness and in the process contributing to the corruption of language, all in order to achieve their goal of more power and prestige.
At least Weeks is pretty honest about his motivation.
But what about the argument that “naturopathic physicians” or “chiropractic physicians” or “acupuncture physicians” can function as primary care practitioners? What sort of argument against this nonsense might actually resonate among not just real physicians but legislators and the public? I can think of one. Just consider the effect on vaccination rates of letting naturopaths, chiropractors, acupuncturists, and all the other non-science-based specialties, the vast majority of which tend to be profoundly anti-vaccine, were allowed to claim the title of “physician” and to function as primary care physicians. Now that’s a truly scary thought.

Please Don’t Define “Complementary and Alternative Health Practices”!
William M. London on October 28, 2011
Link
Since I have a master’s and doctoral degree in health education and since I’m a professor in a department of public health with an undergraduate curriculum that includes substantial attention to health education, I participate in the email discussion group of HEDIR, the Health Education Directory. On August 16th, I received a message to the discussion group from the American Association for Health Education inviting participants to complete an online survey from the Joint Committee on Health Education and Promotion Terminology with results to be analyzed at the Committee’s meeting in September 2011.
The survey items include various terms used by health educators, the currently approved terminology, and three choices followed by a type-in box:
    • This term should remain as defined
    • This term should remain in the report but modified in definition
    • This term is no longer commonly used in health education/health promotion literature
If modify, please provide the suggested wording and reference for that definition if you are citing it from a specific source.
For one of the terms, my preferred response would have be have been a fourth choice that was not offered: The term is commonly used in health education/health promotion and elsewhere, but it should not be used because its use only serves to distort our thought processes and promote quackery.
Here is the term along with the definition presented in the survey:
Complementary and Alternative Health Practices: These practices generally include natural substances, physical manipulations, and self-care modalities. These approaches often incorporate aspects of interventions derived from traditional practices. The approach in Western societies has been to select specific approaches from these systems and apply them to health maintenance, health enhancement, or disease management. Such approaches can be used to compliment[sic] conventional allopathic care (complementary therapy), or as an alternative to conventional approaches (alternative therapy). Many of these complementary and alternative approaches have not been validated through experiential research, but those that have, such as acupuncture for pain, are being integrated into conventional health practices (integrative medicine).
And here are my objections to the term and to the definition given:
“Complementary and alternative health practices” is marketing doublespeak that conceals how promoting (via advertising, publicity, direct selling, word-of-mouth, etc.) non-validated or invalidated practices is unethical. When a practice is science-based, it is simply part of good healthcare or health promotion. “Complementary and alternative” jargon is never necessary to describe validated practices in health promotion or health care delivery. Science-based uses of natural substances, physical manipulations, and self-care modalities are all part of regular medicine.
Science-based natural products medicine is called pharmacognosy. Labels like “complementary and alternative” are used to give the impression of legitimacy, not to pharmacognosy, but to superstitious and often ecologically destructive uses of natural products such as herbalism (particularly paraherbalism), gruesome extractions of bile from living bears, shark cartilage, and rhinoceros horns.
Physical manipulations with a rational basis such as many of those included in personal exercise programs and physical therapy do not require euphemistic labels such as “complementary” or “alternative.” However, the labels “complementary” or “alternative” are often applied to give the appearance of legitimacy to superstition-based or pseudoscience-based physical manipulation treatments such as those used in chiropractic. Many chiropractors falsely claim that the spine requires periodic maintenance “adjustments” of health compromising  “subluxations” that only chiropractors can supposedly detect. Such adjustments don’t complement anything else and they aren’t a viable alternative for health promotion or disease prevention.
Many of the manipulations promoted as “complementary” or “alternative” are actually non-physical; they are rooted in vitalism, which is defined as: “a doctrine that the functions of a living organism are due to a vital principle distinct from physicochemical forces” or “a doctrine that the processes of life are not explicable by the laws of physics and chemistry alone….” Different health cults have different names for the supposed vital principle. In anthroposophy, the names are the divine element in nature, astral body, formative force, or either body. In Ayurvedic medicine, it’s prana. In chiropractic, it’s innate intelligence. In Reichian psychotherapy, it’s orgone energy. In homeopathy, it’s vital energy. In naturopathy, it’s vis medicatrix naturae. In Traditional Chinese Medicine and acupuncture, it’s chi or qi or ki.
The vital principle was popularized in Star Wars as “the force.” But in the real universe, nothing like “the force” is reliably detectable and there are no Jedi-like masters who can manipulate anything akin to it for healing or any other purposes. In the movie Austin Powers: The Spy Who Shagged Me, the vital principle was called mojo. The idea that some people have skills as mojo detectors or mojo manipulators is as absurd as the movie, but “there are some ideas so absurd that only an intellectual could believe them.”
Self-care modalities have been promoted as “complementary” and “alternative,” but what useful distinction is there to be made between supposed “complementary” or “alternative” self-care modalities and those that don’t qualify and are therefore implicitly non-complementary or non-alternative? If the distinction is to be based on validation for safety and efficacy, why introduce euphemistic language like “complementary” or “alternative”? Categories such as validated, non-validated, and invalidated should suffice to give consumers useful information for deciding what modalities of self-care are worth trying out.
It’s true that many tradition-based practices are promoted as “complementary” or “alternative.”  Scholars who attempt to advance “complementary and alternative medicine” often like to emphasize traditional systems of care and ignore other practices marketed as “complementary and alternative.” Since it is often considered rude to be judgmental about traditions associated with particular cultures, fallacious appeals to traditional wisdom are useful in public relations. But numerous practices, products, and services marketed to consumers as “complementary” or “alternative” are promoted as “innovative,” “advanced,” “cutting edge,” “modern,” “scientific,” and the like, not as tradition-based. Examples include such so-called complementary and alternative medicine approaches (sCAMs) as metabolic therapy, chelation therapy, oxygenation treatments, insulin potentiation therapy, clinical ecology, anti-aging medicine, attachment therapy, various other mental health therapies, antineoplastons, cellular therapy, and syncrometers & zappers.
Referring to the selection of specific approaches from traditional systems in Western societies as “complementary” or “alternative” implies an East-West dichotomy that is simply false. Tradition-based systems and supposed whole-system care are not uniquely Eastern. Is it only in Western societies that approaches from traditional systems get used separately from whole-system care?
Medical anthropologists, medical sociologists, educated laypersons, health educators, and even physicians often make the mistake of describing standard medical practices of today and recent decades as “conventional allopathic care.” Allopathy is a term coined by Samuel Hahnemann (formulator of homeopathic treatment principles) as a label for medical practices of his day that were based upon ancient Greek humoral theory of disease such as bleeding and purging and blistering to manipulate the four so-called body humors: blood, phlegm, black bile, and yellow bile. As medicine became more science-based, it discarded treatment based upon the convention of manipulating body humors and progressed by developing healthcare consistent with progress in biological and physical sciences. Nevertheless, approaches to healthcare based upon humoral theory—what Hahnemann called allopathy—persist today in parts of India, Pakistan and elsewhere as Unani medicine, which, ironically, the World Health Organization recognizes as a type of “CAM.” Unani is an Arabic adjective meaning Greek.
Since modern medicine makes progress by relying on science, it is iconoclastic—the antithesis of conventional. By contrast, the real allopathy practiced today as Unani medicine is bound to its ancient conventions. Like much of what gets promoted as “complementary” and “alternative,” Unani medicine reflects conventional wisdom of healing traditions rather than the rigor of scientific testing and the iconoclasm of scientific discovery.
I have previously explained that calling an approach to healthcare “complementary” implies that it adds to the outcome when combined with some other treatment and that calling an approach to healthcare “alternative” implies that it can be successfully used in lieu of some other approach. However, this is misleading labeling. Simply calling an approach “complementary” doesn’t mean it actually complements anything else and calling an approach “alternative” doesn’t make it a viable alternative. The jargon “complementary and alternative” serves to distract attention away from questions of utility based upon scientific merit.
Professor Richard Dawkins has explained: “Either it is true that a medicine works or it isn’t. It cannot be false in the ordinary sense but true in some ‘alternative’ sense.”
Drs. John E. Dodes and Marvin Schissel put it this way: “Erythromycin is an alternative to penicillin, but a pogo stick is not an alternative to an automobile.”
Drs. Marcia Angell and Jerome Kassirer wrote: “There cannot be two kinds of medicine—conventional and alternative. There is only medicine that has been adequately tested and medicine that has not, medicine that works and medicine that may or may not work.”
Dr. George Lundberg explains it this way:
There is no “alternative medicine.” There is only medicine:
    • Medicine that has been tested and found to be safe and effective. Use it; pay for it.
    • And, medicine that has been tested and found to be unsafe or ineffective. Don’t use it; don’t pay for it.
    • And, medicine for which there is some plausible reason to believe that it might be safe and effective. Test it and then place it into one of the other two categories.
Although many people believe that acupuncture for pain is medicine that fits Dr. Lundberg’s first category,  the weight of evidence places it in the second category, especially considering the lack of a plausible rationale for acupuncture as a therapy. Few, if any, health practices that have been promoted as “complementary” and “alternative” also belong in Dr. Lundberg’s first category. More than ten years of research funding by the National Center for Complementary and Alternative Medicine has failed to contribute to medical progress.
The term integrative medicine is superfluous and should not be used by responsible health professionals. Palliative care and adjunctive care are meaningful and useful terms for efforts to provide rational modalities of humane care, comfort, and support addressing the diverse needs of patients. The term “integrative medicine” adds nothing to describe approaches strongly supported by scientific evidence, but serves as an income-generating mechanism for attracting patients to seemingly special modalities that typically lack support beyond cherry-picked evidence or tradition. The term “integrative medicine” is not needed to offer science-based psychological approaches for managing health problems, but it does help in marketing when you are offering modalities based on vitalism. “Integrative medicine” represents branding, not a meaningful medical specialty. It projects a misleading image of academic seriousness that serves only to obscure its hype and help secure funding for clinical research of dubious need.
Terms such as “alternative,” “complementary,” and “integrative” have become popular euphemisms for non-validated and invalidated approaches to health enhancement—especially approaches with farfetched rationales. The use of such euphemisms facilitates quackery: the promotion of health products, services, or practices of questionable safety, effectiveness, or validity for an intended purpose. Today quackery is a far less popular term than the euphemisms. In some circles, it is politically incorrect to refer to quackery. But if we cannot refer to quackery as quackery, we can expect it to persist as a neglected public health scandal. I suggest that there are better alternatives to using currently popular euphemisms of alt-speak.

The difference between science-based medicine and CAM
There is a huge difference between science-based medicine (SBM) and so-called “complementary and alternative medicine” (CAM) or, as it’s increasingly called, “integrative medicine.” That difference is that SBM changes with new science. The change might be messier and slower than we would like, but eventually science and evidence win out.
David Gorski on July 29, 2013
Link
“Alternative medicine,” so-called “complementary and alternative medicine” (CAM), or, as it’s become fashionable to call it, “integrative medicine” is a set of medical practices that are far more based on belief than science. As Mark Crislip so pointedly reminded us last week, CAM is far more akin to religion than science-based medicine (SBM). However, as I’ve discussed more times than I can remember over the years, both here and at my not-so-super-secret-other blog, CAM practitioners and advocates, despite practicing what is in reality mostly pseudoscience-based medicine, crave the imprimatur that science can provide, the respect that science has. That is why, no matter how scientifically implausible the treatment, CAM practitioners try to tart it up with science. I say “tart it up” because they aren’t really providing a scientific basis for their favored quackery. In reality, what they are doing is choosing science-y words and using them as explanations without actually demonstrating that these words have anything to do with how their favored CAM works.
A more important fundamental difference between CAM and real medicine is that CAM practices are not rejected based on evidence. Basically, they never go away. Take homeopathy, for example. (Please!) It’s the ultimate chameleon. Even 160 years ago, it was obvious from a scientific point of view that homeopathy was nonsense and that diluting something doesn’t make it stronger. When it became undeniable that this was the case, through the power of actually knowing Avogadro’s number, homeopaths were undeterred. They concocted amazing explanations of how homeopathy “works” by claiming that water has “memory.” It supposedly “remembers” the substances with which it’s been in contact and transmits that “information” to the patient. No one’s ever been able to explain to me why transmitting the “information” from a supposed memory of water is better than the information from the real drug or substance itself, but that’s just my old, nasty, dogmatic, reductionist, scientific nature being old, nasty, dogmatic, reductionist, and scientific. Then, of course, there’s the term “quantum,” which has been so widely abused by Deepak Chopra, his acolytes, and the CAM community in general, while the new CAM buzzword these days to explain why quackery “works” is epigenetics. Basically, whenever a proponent of alternative medicine uses the word “epigenetics” or “quantum” to explain how an alternative medicine treatment “works,” what he really means is, “It’s magic.” This is a near-universal truth, and even the most superficial probing of such justifications will virtually always reveal magical thinking combined with an utter ignorance of the science of quantum mechanics or epigenetics.
So, yes, much of CAM is either very much more like religion than science in that CAM is immune to evidence. True, the scientific “explanations” change, and CAM practices might evolve at the edges based on evidence, but the core principles remain. You don’t see, for example, homeopaths or naturopaths deciding that homeopathy doesn’t work because science and clinical trials overwhelmingly show that it is nonsense. You don’t see chiropractors leaving chiropractic in droves because they’ve come to the realization that subluxations don’t exist and they can’t cure allergies, heart disease, gastrointestinal ailments (or anything else) but rather are in reality physical therapists with delusions of grandeur. Ditto reiki, acupuncture, therapeutic touch, and “energy healing.” These practices persist despite overwhelming evidence that they do not work and are based on magical thinking, not science. All of the scientific studies and clinical trials funded by NCCAM and other CAM-friendly organizations never actually take the next step from all the negative studies of CAM and come to the conclusion that they should stop using such modalities.
The key difference between SBM and CAM
No one is saying that the record of SBM is perfect when it comes to changing nimbly with new evidence, and any imperfection in the record of SBM and evidence-based medicine (EBM) actually being, well, science- and evidence-based, is a favorite target of CAM apologists. Hence there are frequent claims circulating that only 15% of medicine is actually evidence-based. It’s a bogus claim, a myth, as Steve Novella has pointed out. In reality, studies appear to converge on estimates that approximately 80% of interventions are based on compelling evidence, and between 30-60%, depending on the specialty, are based on randomized clinical trials. That’s not good enough, but it’s far better than CAM apologists would lead you to believe, and it’s certainly far better than anything in CAM.
Nonetheless, it has been recognized for a long time that EBM/SBM is sometimes slow to change in response to new evidence. Indeed, there was an aphorism I heard while in medical school that outdated treatments and procedures don’t die off completely until the physicians who learned them during residencies or fellowships die off. I learned that that’s not entirely true. There is, after all, a gap of around 20 years between the time a generation of physicians retires and dies off; so such practices actually die off much sooner. I keed, I keed, of course, but the point is valid.
There is the opposite problem in EBM/SBM as well, namely a tendency towards a “bandwagon” effect wherein a new therapy is widely adopted before there is solid evidence of its superiority (or at least of its non-inferiority with alternate benefits). I’m a surgeon, so I know that, unfortunately, the surgical world is very much prone to this sort of problem. Surgeons tend to like shiny, pretty new toys and to do spiffy new procedures that prove that they are the biggest, baddest scalpel cowboys in the all the land. These tendencies have led to a number of procedures becoming widely adopted before they were definitely shown to be superior. Laparoscopic cholecystectomy is the example that I like to use the most; it swept the surgical world over 20 years ago without compelling evidence for its safety. Later, it was found that the incidence of common bile duct injury was much higher after laparoscopic cholecystectomy than conventional cholecystectomy. That incidence fell as more surgeons became more facile at the procedure, but it was years before there was compelling evidence that the laparoscopic approach was truly superior. History seems to be repeating itself today with robotic surgery. At the risk of offending some of my surgical colleagues, I’ve yet to see compelling evidence that doing, for example, a radical prostatectomy with the da Vinci robot is truly superior to doing it using what was the new way ten or fifteen years ago but is now the old way, using laparoscopy. From my perspective evaluating existing evidence, the da Vinci is as safe and effective as laparoscopy, but if it is sufficiently more so to justify its much greater cost I haven’t seen the evidence yet. I sometimes joke that if it were possible to do breast surgery (my specialty) with the da Vinci, then I’d be all for it. Maybe I’ll have to look into that. I could be bigger than Armando Guiliano, and time’s wasting. I probably only have 15 or 20 years left in my career to make an international name for myself.
But how often are medical practices found to be ineffective and abandoned? How much do we test existing practices in light of new data? There have been a number of studies looking at this issue, which is already a marked contrast to CAM, where ineffective practices are, as far as I can tell, never abandoned. The most recent of these caught my eye last week. Published in the Mayo Clinic Proceedings by a team from the National Cancer Institute, the University of Chicago (one of my alma maters!), Northwestern University, George Washington University, and Lankenau Medical Center and entitled A Decade of Reversal: An Analysis of 146 Contradicted Medical Practices, this study seeks to get a handle on the answer to that very question for these reasons:
We expect that new medical practices gain popularity over older standards of care on the basis of robust evidence indicating clinical superiority or noninferiority with alternative benefits (eg, easier administration and fewer adverse effects). The history of medicine, however, reveals numerous exceptions to this rule. Stenting for stable coronary artery disease was a multibillion dollar a year industry when it was found to be no better than medical management for most patients with stable coronary artery disease.1 Hormone therapy for postmenopausal women intended to improve cardiovascular outcomes was found to be worse than no intervention,2 and the routine use of the pulmonary artery catheter in patients in shock was found to be inferior to less invasive management strategies.3 Previously, we have called this phenomenon (when a medical practice is found to be inferior to some lesser or prior standard of care) a medical reversal.4, 5, 6 Medical reversals occur when new studies—better powered, controlled, or designed than their predecessors—contradict current practice.4 In a prior investigation of 1 year of publications in a high-impact journal, we found that of 35 studies testing standard of care, 16 (46%) constituted medical reversals.4 Another review of 45 highly cited studies that claimed some therapeutic benefit found that 7 (16%) were contradicted by subsequent research.7
Identifying medical practices that do not work is necessary. The continued use of such practices wastes resources, jeopardizes patient health, and undermines trust in medicine. Interest in this topic has grown in recent years. The American Board of Internal Medicine launched the Choosing Wisely campaign,8 a call on professional societies to identify the top 5 diagnostic or therapeutic practices in their field that should not be offered.9 In England, the National Institute for Health and Clinical Excellence has tried to “disinvest” from low-value practices, identifying more than 800 such practices in the past decade.10 Other researchers have found that scanning a range of existing health care databases can easily generate more than 150 low-value practices.11 Medical journals have specifically focused on instances in which more health care is not necessarily better. The Archives of Internal Medicine created a new feature series in 2010 entitled “Less is More.”12
One can’t help but note right from the introduction of this paper that SBM/EBM does continually reevaluate its practices and treatments, testing which ones work and which ones do not and comparing current practice against new treatments. Granted, the intensity of this effort seems to be a more recent development, with the implementation of the Patient Protection and Affordable Care Act, but is it really? This article suggests that the answer is: perhaps not.
The authors specifically examine the question of how much of the medical literature consists of what they refer to as “medical reversals,” as described above. Specifically, they tried to estimate what percentage of the medical literature consists of articles that question current medical practice, particularly that consist of high quality evidence suggesting that current practice needs to be changed or that a standard-of-care intervention doesn’t work, doesn’t work as well as a non-standard-of-care intervention, or is actually harmful. How the authors did this, I find easier to let them describe:
Two reviewers (C.T., A.V., M.C., J.R., S.Q., S.J.C., D.B., V.G., or S.S.) and V.P. read articles addressing a medical practice in full. On the basis of the abstract, introduction, and discussion, articles were classified as to whether the practice in question was new or existing. Methods were classified as one of the following: randomized controlled trial, prospective controlled (but nonrandomized) intervention study, observational study (prospective or retrospective), case-control study, or other methods. End points for articles were classified into those that reached positive conclusions and those that found negative or no difference in end points. Lastly, articles were given 1 of 4 designations. Replacement was defined as a new practice surpassing an older standard of care. Back to the drawing board was defined as a new practice failing to surpass an older standard. Reversal was designated when a current medical practice was found to be inferior to a lesser or prior standard. Reaffirmation was defined as an existing medical practice being found to be superior to a lesser or prior standard. Finally, articles in which no firm conclusion could be reached were termed inconclusive. The designation of an article was also performed in duplicate. When there were differences in opinion between the 2 reviewers, adjudication first involved discussion between the 2 readers to see whether agreement could be reached. If disagreement persisted, a third reviewer (A.C.) adjudicated the discrepancy. Less than 3% of articles required discussion, and less than 1% required adjudication. A table detailing each medical reversal was constructed (Supplemental Appendix; available online at http://www.mayoclinicproceedings.com), and the third reviewer (A.C.) reviewed all reversals.
So what did the investigators (Prasad et al) find? They examined ten years’ worth of NEJM original reports, from 2001 through 2010, for a total of 2,044 original articles. Of these, 1,344 (65.8%) addressed a medical practice, of which 911 (68%) were randomized controlled trials, 220 (16%) were prospective controlled but non-randomized studies, 117 (9%) were observational studies, 43 (3%) were case-control studies, and 53 (4%) used other methods. Of these 1,344 reports, 981 (73%) studied a new medical practice, while 363 (27%) addressed an existing practice. Overall, 756 articles (56%) found that a new practice surpassed the existing standard of care at the time (replacement), while 165 (12%) failed to find that a new practice was better than existing practices. In terms of what we’re really interested in, of the 363 studies examining an existing practice, 146 studies (40%) were reversals, while 138 (38%) upheld standard practices. Here’s a breakdown from the article for your edification:
Of the reversal articles, not surprisingly most (76%) turned out to be randomized clinical trials, and interestingly, the percentage of each type of trial didn’t change much over the decade-long study period:
The one problem I had with this study was that it only looked at one journal: The New England Journal of Medicine. I can understand why the authors might have chosen that particular journal. It’s very high impact, and, with the exception of a recent distressing tendency to let some low quality CAM articles slip in, one of the more rigorous medical journals out there that isn’t a specialty journal; i.e., it accepts articles covering all areas of medicine. It’s not a basic science journal; it generally only publishes original studies that are either clinical trials, epidemiological studies, or at the very least highly translational. It also, from my reading, only rarely publishes really preliminary clinical work, such as phase I clinical trials. On the other hand, one has to wonder whether the results would be generalizable to the rest of the medical literature.
For example, according to this study, articles in the NEJM that tested new practices were far more likely to find them beneficial than articles that tested existing ones (77.1% vs 38.0%), while articles that tested existing standard-of-care practices were far more likely to find those practices ineffective than articles testing new practices (40.2% vs 17.0%). Looking at such numbers, I can’t help but wonder if there is a publication bias for finding new therapies effective and/or for finding existing therapies either ineffective or harmful, particularly in the NEJM, which is among the highest of high-impact medical journals. Think about it. Who thinks that their findings are substantial enough and interesting enough to be seriously considered for publication in the NEJM? It’s investigators who have found that some new therapy works for a common or very serious disease, but it wouldn’t surprise me if it’s also authors who have found compelling evidence that a commonly used existing standard of care is either not effective or is even dangerous.
It’s also informative to look at some of the medical practices that were the subject of reversal articles. For instance, it was thought that certain vaccinations could increase the risk of relapse in multiple sclerosis, but two studies showed no increased risk. One looked at tetanus, hepatitis B, and influenza vaccination; the other at hepatitis B vaccination. One showed that delayed drainage of effusion in otitis media did not result in worse outcomes than immediate placement, resulting in a change in practice. Another key reversal came in the form of a 2003 study that showed that high-dose chemotherapy followed by bone marrow transplantation did not improve survival in advanced breast cancer. This was a huge one, and almost immediately oncologists stopped doing bone marrow transplants for breast cancer. Another showed that the use of pulmonary artery catheters in acute lung injury didn’t improve outcomes and was associated with more complications. (When I was a resident in the 1990s, all of these patients got pulmonary artery catheters.) A couple of these I’ve written about, such as vertebroplasty. More recently, there was a study that showed no benefit to routine PSA screening for prostate cancer in American men.
Indeed, I can’t help but mention here that the whole reevaluation of routine screening for cancer, such as PSA screening for prostate cancer and mammography for breast cancer, topics I’ve written about numerous times for this blog, are examples of exactly that: SBM/EBM evaluating current practices in light of new data and determining whether they should be changed or abandoned. Routine PSA screening for men at average risk of prostate cancer has more or less been abandoned, for example, while current mammography practices are being questioned as promoting too much overdiagnosis and likely will evolve in response.
Choosing wisely
Perhaps the most prominent example of the efforts EBM/SBM makes to continually reevaluate its practices is the Choosing Wisely initiative. Scott Gavura brought it up last year, and I’ve mentioned it elsewhere. It’s basically an effort by organized medicine to reduce the use of what are known as “low value” tests (i.e., tests that provide little or no benefit but are often costly and can produce complications and more invasive testing). If screening tests are a problem, there are also a lot of tests that are ordered too frequently or for dubious indications. The reasons can range from laziness to defensive medicine, but whatever the reason, such tests cost money, can lead to incidental findings that need further workup, and can even lead to overdiagnosis. In 2010, Dr. Howard Brody published a challenge to his physician colleagues in The New England Journal of Medicine. It was an amazing article, in which Dr. Brody challenged physician specialty organizations thusly:
In my view, organized medicine must reverse its current approach to the political negotiations over health care reform. I would propose that each specialty society commit itself immediately to appointing a blue-ribbon study panel to report, as soon as possible, that specialty’s “Top Five” list. The panels should include members with special expertise in clinical epidemiology, biostatistics, health policy, and evidence-based appraisal. The Top Five list would consist of five diagnostic tests or treatments that are very commonly ordered by members of that specialty, that are among the most expensive services provided, and that have been shown by the currently available evidence not to provide any meaningful benefit to at least some major categories of patients for whom they are commonly ordered. In short, the Top Five list would be a prescription for how, within that specialty, the most money could be saved most quickly without depriving any patient of meaningful medical benefit. Examples of items that could easily end up on such lists include arthroscopic surgery for knee osteoarthritis and many common uses of computed tomographic scans, which not only add to costs but also expose patients to the risks of radiation.
Some specialty organizations have done just that, and the result is called Choosing Wisely. To begin, nine specialty societies have produced lists of Five Things Physicians and Patients Should Question, which Choosing Wisely describes as “evidence-based recommendations that should be discussed to help make wise decisions about the most appropriate care based on a patients’ individual situation.” The clinical societies that have participated include:
    • American Academy of Allergy, Asthma & Immunology
    • American Academy of Family Physicians
    • American Academy of Hospice and Palliative Medicine
    • American Academy of Neurology
    • American Academy of Ophthalmology
    • American Academy of Otolaryngology — Head and Neck Surgery Foundation
    • American Academy of Pediatrics
    • American College of Cardiology
    • American College of Obstetricians and Gynecologists
    • American College of Physicians
    • American College of Radiology
    • American College of Rheumatology
    • American Gastroenterological Association
    • American Geriatrics Society
    • American Society for Clinical Pathology
    • American Society of Clinical Oncology
    • American Society of Echocardiography
    • American Society of Nephrology
    • American Society of Nuclear Cardiology
    • American Urological Association
    • Society for Vascular Medicine
    • Society of Cardiovascular Computed Tomography
    • Society of Hospital Medicine – Adult Hospital Medicine
    • Society of Hospital Medicine – Pediatric Hospital Medicine
    • Society of Nuclear Medicine and Molecular Imaging
    • The Society of Thoracic Surgeons
It’s an impressive list. Naturally, being a cancer surgeon, I can’t resist looking here at the recommendations made by the American Society of Clinical Oncology (ASCO). Interestingly, two out of the five recommendations were breast cancer-related:
    • Don’t perform PET, CT, and radionuclide bone scans in the staging of early breast cancer at low risk for metastasis.
    • Don’t perform surveillance testing (biomarkers) or imaging (PET, CT, and radionuclide bone scans) for asymptomatic individuals who have been treated for breast cancer with curative intent.
It’s true. Far too often we do a million-dollar workup for patients with early stage breast cancer, and there is pretty much zero good evidence that these workups improve survival, improve care, or otherwise do anything except cost a lot of money, delay definitive treatment, expose the patient to radiation, and provoke worry in both patient and practitioner. I realize this is anecdotal experience, but overuse of these tests in early stage breast cancer doesn’t appear to as much of a problem in big cancer centers as it does in community cancer hospitals. But it is a problem in a lot of places.
One thing that disappointed me was that the Choosing Wisely list from the American College of Radiology didn’t appear to include any breast cancer-related recommendations. For instance, routine breast MRI before surgery for breast cancer increases the rate of mastectomy and, worse, contrary to the stated intent for preoperative MRI, does not decrease the rate of reexcision. On the other hand, it’s refreshing to see a recommendation that most surgeons instinctively know to be true:
Avoid admission or preoperative chest x-rays for ambulatory patients with unremarkable history and physical exam.
Performing routine admission or preoperative chest x-rays is not recommended for ambulatory patients without specific reasons suggested by the history and/or physical examination findings. Only 2 percent of such images lead to a change in management. Obtaining a chest radiograph is reasonable if acute cardiopulmonary disease is suspected or there is a history of chronic stable cardiopulmonary disease in a patient older than age 70 who has not had chest radiography within six months.
Of course, although doctors carry a large share of the responsibility for unnecessary tests, medications, and care, they are not alone in being responsible for the overuse of various medical tests and interventions. The classic example, of course, is the use of antibiotics for viral infections, which is something that patients often demand in the mistaken belief that it will help them and that many doctors use because patients ask for it and it’s easier to give in than to spend the time it takes to explain why it’s not medically indicated. Another contributor to the problem is that many of these tests have a significant financial incentive. It’s the same problem that contributes to the slowness of the decline in use of some treatments shown not to be effective.
Lest any reader think that I don’t put my money where my mouth is, I will mention that I was recently appointed co-director of a state-wide program designed to increase quality of breast cancer care by promoting adherence to evidence-based guidelines and one of our primary goals this year will be to promote adherence among our member hospitals to the Choosing Wisely guidelines.
We change, but it’s messy and slow
One reason why EBM/SBM is slower than we might like to eliminate outdated and ineffective practices is simple. It’s not easy. Evidence from science, epidemiology, and clinical trials takes a long time to come in. It’s often very messy. When a practice comes into question, there will often be conflicting evidence, and it often takes a number of studies before conclusions about the practice firm up to the point where they are incorporated into evidence-based guidelines and become standard of care.
Often, practices that are later reversed come into usage based on premature and inadequate evidence. Often, small trials look promising, and physicians start using a treatment based on them. Sometimes such practices become standard based on short term outcome measures, and when long term data become available previously unsuspected harms become apparent. Sometimes it’s excessive confidence in the appropriateness of the proposed mechanism used to explain why the treatment should work. What is needed, according to Prasad et al (and I agree), is more rigor:
As such, we favor policies that minimize reversal. Nearly all such measures involve raising the bar for the approval of new therapies6, 83, 84 and asking for evidence before the widespread adoption of novel techniques. In all but the rarest cases,82 large, robust, pragmatic randomized trials measuring hard end points (with sham controls for studies of subjective end points) should be required before approval or acceptance. Our position is in contrast to efforts to lower standards for device and drug approval,85 which further erodes the value of the regulatory process.
One can’t help but note that this is in marked contrast to CAM studies, in which CAM advocates ask us to accept much less rigorous types of evidence to accept modalities. As Steve Novella has frequently pointed out, as rigorous randomized clinical trials show that most CAM interventions are no better than placebo, the refrain we frequently hear is that we should look at “pragmatic” trials. In this context, pragmatic doesn’t mean the same thing. What Prasad et al are referring to are randomized trials that reflect real-world practices. What I mean by “pragmatic” trials in the context of acupuncture are more observational trials of how the treatment is used in the real world. As I’ve said many times, this is putting the cart before the horse. Normally pragmatic trials are done for treatments that have already been shown to be efficacious in randomized clinical trials. They can’t show efficacy by themselves. They are designed to test how treatments already shown to be efficacious in randomized trials function once let “out into the wild” (i.e., the real world). Frequently, outside the rarified, rigorous world of randomized clinical trials, treatments are less effective.
It should also be pointed out that, just because a treatment was “reversed” in a clinical trial doesn’t necessarily mean that the older practices reversed were wrong. However, as Prasad et al put it:
The reversals we have identified by no means represent the final word for any of these practices. Simply because newer, larger, better controlled or designed studies contradict standard of care does not necessarily mean that older practices are wrong and new ones are right. On average, however, better designed, controlled, and powered studies reach more valid conclusions.94 Nevertheless, the reversals we have identified at the very least call these practices into question. Some practices ought to be abandoned, whereas others warrant retesting in more powerful investigations. One of the greatest virtues of medical research is our continual quest to reassess it.
So, yes, “conventional” medicine doesn’t always get it right. Occasionally it gets it wrong, on rare occasions spectacularly wrong. But unlike most CAM modalities, EBM/SBM is self-correcting. It actually does abandon treatments that don’t work. The process might be messy and ugly at times, but it does happen. For example, many years ago, angina pectoris was sometimes treated with a surgical procedure known as mammary artery ligation. The idea was that tying off these arteries would divert more blood to the heart. The operation became popular on the basis of relatively small, uncontrolled case series. Then, two randomized, sham surgery-controlled clinical trials were published in 1959 and 1960. Both of these trials showed no difference between bilateral internal mammary artery ligation and sham surgery. Very rapidly, surgeons stopped doing this operation. A similar example is one I mentioned above: bone marrow transplantation for advanced breast cancer, which was similarly rapidly abandoned after randomized clinical trials showing it to be no better than the previous standard of care. I’m not saying that this happened without conflict or disagreement; proponents of these therapies can always find reasons to discount the clinical trial evidence. But in the end evidence and science do win out.
Now compare this to CAM practices. Can anyone name a CAM treatment that was abandoned by CAM practitioners as a result of research and randomized clinical trials showing that it doesn’t work? A single one? I can’t. That’s the difference between CAM and EBM/SBM. The day that I see a CAM practice go extinct, like bilateral internal mammary artery ligation for angina pectoris, is the day that I might start to take seriously CAM practitioner claims that they are science-based.

“Integrative” medicine versus “alternative” medicine
David Gorski on May 15, 2016
Link
I’ve written a lot about the language issue with respect to alternative medicine. As I like to put it (at least in shortened form), first there was quackery. Quacks did not like that name at all, and thus was born alternative medicine. And the quacks did think it good—for a while. There was a problem, however. “Alternative” medicine implied (correctly, of course) that what was being discussed was not real medicine, and the quacks could not abide that. Thus was born “complementary and alternative medicine” (CAM).
And the quacks thought this very good indeed.
Unfortunately, it was not long before the problem with the term CAM became apparent. It had the word “complementary” in it. The implication of that word, of course, is that what they were doing was still somehow not real medicine. It was complementary to real medicine, the icing on the cake, if you will. Real medicine could do without it, and having that implication in the very name that their evolving specialty had taken on was offensive to the quacks.
So they changed it.
Thus was born “integrative medicine.” This was a term that eliminated all implication that quackery such as reiki, acupuncture, naturopathy, and the rest of the pseudoscience and antiscience that made up the bulk of the “alternative” medicine in complementary and alternative medicine was somehow lesser medicine. That it is lesser medicine—or not even medicine at all in most cases—bothered the quacks not at all. It was all about marketing, and “integrative medicine” implied a meeting of equals. More importantly, it could be sold as the “best of both worlds,” which sounds great until you consider that one world consists of medicine that works and the other world consists largely of magic. Few thought of that, however, and the term “integrative medicine” had the added bonus of being so amorphous that it could “integrate” just about anything, even what might or might not have been underutilized areas of real, science-based medicine, such as nutrition, diet, and exercise.
Thus, the term “integrative medicine” not only eliminated the stigma of being “alternative” or “complementary,” but it also facilitated the rebranding of perfectly science-based areas of medicine (like the aforementioned diet and exercise) as somehow being “alternative” or somehow unique to integrative medicine. It was a masterstroke of marketing, because diet, exercise, and lifestyle interventions can actually work (that is, when they’re not distorted beyond recognition by woo, which is part of integrative medicine) and therefore lend some of their legitimacy to the serious woo that has been “integrated” with them in the specialty known as integrative medicine.
But not everyone in the world of “integrative” medicine is happy about this state of affairs. John Weeks, for instance, just wrote a post in that wretched hive of scum and quackery, The Huffington Post, entitled “In Defense of ‘Alternative Medicine’.” Using the recent BMJ op-ed (no, contrary to how it’s portrayed in the press, it was most definitely not a study and did not add any new knowledge to the scientific literature) estimating there to be a quarter of a million deaths due to medical error, Weeks tries to co-opt the word “alternative” in arguing for a bit of a return of the word to describe what he has long promoted. It’s also a reversal of sorts, given that he was very much into promoting the term “integrative medicine” not too long ago.
Basically, Weeks co-opts the word “alternative” by conflating the use of the word with respect to medicine and with respect to the bad things he sees in medicine. It’s essentially a variant of the sort of false dichotomy I rail against not infrequently, the dichotomy that claims that to “fix” what’s wrong with “conventional” medicine one has to embrace quackery. Here’s a taste:
No — we sought and still seek alternatives to dehumanizing, production-centered, technology-oriented, natural process-denying, bulling disregard for the value of other professions and practices that an alternative “holistic” or “whole person” or “natural” or “mind-body” approach to what ailed us.
No one could argue that there are aspects of the way medicine is practiced right now can at times be dehumanizing. However, it is entirely possible to practice “holistic” medicine without embracing the quackery of alternative medicine. Similarly, there’s a reason why science-based practitioners have such “bulling disregard for the value of” all those “holistic” professions. It’s because most of them are rooted in pseudoscience and practice quackery (e.g., naturopathy). Respect is earned, and what earns respect among physicians is excellent patient care based on the most rigorous science.
Next up are a several demands that mix the reasonable with apologetics for quackery:
We want an alternative to an opioid strategy such as that propounded by the CDC and the Obama administration’s 1.1-billion plan that denies the value of multiple non-pharmacologic, alternative and integrative approaches.
We want an alternative that says Never Only Opioids.
We want an alternative that says never only any long-term course of a drug known to have significant adverse consequences when used chronically if other approaches might limit the need or reduce the harm.
We want an alternative that focuses on health, well-being and healing.
The Obama administration’s plan doesn’t “deny the value of multiple non-pharmacologic, alternative and integrative approaches”; it focuses on strategies that have some science behind them supporting their efficacy and safety. I also note that, through his link, Weeks tries to co-opt the American Public Health Association’s (APHA) call for “provider training programs on mental health, non-pharmacological pain treatment alternatives, substance abuse and overdose prevention.” Unfortunately, the APHA does fall for the pseudoscience behind integrative medicine by suggesting “complementary and integrative care providers.” It irritates me to no end when what should be science-based professional societies fall for the message of alternative medicine providers.
Be that as it may, here’s the language sleight-of-hand going on here. When physicians say “non-pharmacological strategies” to deal with anything, they mean exactly that: strategies that don’t involve pharmaceutical drugs. However, they also mean science-based strategies that don’t involve drugs. That implication is unstated but there. When Weeks uses the term “non-pharmacological strategies,” what he means is alternative medicine, CAM, “integrative medicine,” or whatever you want to call the various forms of quackery that “integrative medicine” integrates.
He soon makes this clear:
We want an alternative to decision processes that exclude entire licensed practitioner groups — chiropractors, acupuncturists, integrative and naturopathic doctors, holistic nurses, massage and yoga therapists — when it is clear that millions of citizens of the United States are benefiting from their services every single day. Right now. This moment. Not awaiting Big Brother’s sanction.
There! That’s Weeks’ message, his real message. He wants to let the quacks in: chiropractors, acupuncturists, integrative and naturopaths. True, massage therapy doesn’t have to be quackery (although unfortunately massage therapists are prone to injecting a lot of woo into their ministrations, particularly “energy medicine”). Neither does yoga, at least when stripped of its mystical, vitalistic components, such as Kundalini. Stripped of Kundalini, yoga becomes nothing more than a form of exercise.
To confuse things further, Weeks “integrates” some reasonable “alternatives” along with his wanting to integrate alternatives that consist of quackery. His language, of course, is full of hyperbole, but the basic message, stripped of that hyperbole, is not too hard to get on board with:
We want an alternative to the 5 minute or 15 minute office calls that drive physicians to prescribe drugs and flat-out deny the complexity of what got people in trouble in the first place, and the time needed to learn, and to unravel and resolve the issues.
Elsewhere:
We want an alternative to a payment system that puts the workaday wizards of change agency with people’s health processes at the bottom of the totem pole or, more likely, hatchets them off altogether.
We want an alternative to an industrial approach to payment and delivery that directly drives the morbidity and mortality reflected in those ugly data from the British Medical Journal. First, do no harm.
Who likes five or fifteen minute office visits for complex patients? Not very many primary care doctors that I know. Who likes the “industrial approach” to payment and delivery? Again, not very many doctors I know, if any. Here comes the false dichotomy again, though. The implication is that if you are not in favor of “integrative medicine,” you must be in favor of five minute office visits, reimbursement that doesn’t always cover expenses, and a dehumanized health care system. It’s nonsense, of course. Fixing the system doesn’t require embracing pseudoscience.
Weeks, however, wants you to think that it does, as he cleverly drops little bombs like these in:
We want an alternative to any medical doctors or other health professionals who lack a primary respect for the roles of nutrition, self-care, mindfulness, sleep, movement, environment, community and the actual science of supporting individuals to health.
Again, no good physician lacks a respect for these things. What he does lack respect for are modalities that have no science behind them, like reiki, acupuncture, reflexology, or most of naturopathy—and rightly so. No physician doesn’t want to support their patients’ journey to health or to help patients remain healthy. Again, supporting patients’ health does not require embracing quackery or abandoning science.
Of course, that’s not Weeks is about:
We want an alternative, as Group Health Research Institute senior investigator Dan Cherkin, PhD recently argued to a still dominant focus of resources on reductive biomedical research and practice.
Of course, Weeks (and Cherkin) argue for getting rid of that pesky “reductionistic” science because it doesn’t support the efficacy of the quackery they want to “integrate” with medicine. It’s an old story. When science doesn’t support you, you start complaining about “reductionism” and then decrease the rigor of the science by combining interventions and getting rid of controls. If “reductionistic” science shows that, for instance, acupuncture and sham acupuncture are indistinguishable, it doesn’t mean that acupuncture doesn’t work, even though the same result for a real medical intervention would mean that intervention doesn’t work. Instead, it must mean we’re harnessing the power of placebo or that “complex interactions” are helping patients. In reality, such “explanations,” such as they are, boil down to nothing more than making excuses.
Weeks concludes by characterizing the “complementary” part of CAM as a “suck-up gesture, a way to “play inside the power-centers of the juggernauts, world views, and abiding priorities that daily generate the BMJ outcomes.” Of course, we who argue for science-based medicine and against the integration of quackery into medicine knew that all along. I’ve said as much. Language is the tool used by quack apologists and quacks to integrate their quackery into medicine. Even so, oddly enough Weeks actually likes the term “complementary” more than he does “integrative”:
The reframing to “integrative” was tap-rooted in the work of medical doctors coming from the alternative and natural medicine traditions, like Andrew Weil, MD. They were beginning to engage the daunting political-economic-medical work of reforming academic medicine from the inside. Diplomacy was required. “Alternative medicine” was a bold challenge. “Integrative” was reassuring. Regular medicine is still the center. You are still in the driver’s seat. The word has been a door-opener.
Yet these integrative medical doctor change agents in their self-definition call for a relationship centered system that is inclusive of all health professionals and disciplines, focused on health and healing. Would you say this is an “alternative” to what we have today?
“Integrative” has its value. But does it capture what is needed?
Maybe it’s because I’m a doctor, but I’ve always viewed the reframing of alternative medicine to CAM to “integrative” as the most effective single use of language used by people like Andrew Weil and John Weeks to persuade physicians that they really were “integrating” the “best of both worlds” into medicine and that the pseudoscience would be left out. Of course, the pseudoscience was not left out; rather the association between modalities like diet or exercise and the pseudoscience facilitated the integration of quackery even as quacky as reiki and other “energy healing” with medicine.
Finally, of course, contrary to Weeks’ assertion, integrative medicine will not decrease medical errors. The more rigorous application of science to the process of delivering care, rather than to just the discovery of new treatments, is what will. It won’t be easy as it is, and certainly integrating quackery with medicine won’t make it any easier, despite what Weeks wants you to think.

Clinical Decision Making: Part I
How do doctors make clinical decisions to arrive at a diagnosis? Part 1 of a 3-part series.
Steven Novella on March 6, 2013
Link
I practice in a university clinic which functions partly as a tertiary referral center, which means we get referrals from other specialists. I also get many referrals for second opinions. Sometimes the entire cause for the patient’s desire for a second opinion, it seems to me, is the simple fact that they did not understand the reasoning of the previous specialist. They were given a diagnosis and a course of treatment, but not an explanation of how their doctor arrived at those conclusions.
I am not being judgmental – different practices are under different pressures and time constraints, and it can be very difficult to gauge a patient’s understanding. Often the physician and the patient are proceeding based upon differing assumptions and narratives that are not expressly stated. The doctor may think they have explained the situation entirely, but simply did not confront misleading assumptions they were not aware their patient had.
This is part of the advantage of engaging the public about health issues and confronting pseudoscience, myths, and misconceptions – you develop a deep awareness of how the general public thinks about medicine.
Carl Sagan warned against scientists becoming a “priesthood” with inaccessible knowledge and jargon, and this pertains to the applied science of medicine as well (perhaps especially). We need to engage with the public so that there is a general understanding of not only the findings of science but how science works. In the context of medicine, this means understanding clinical decision making – how doctors think.
Toward that end I plan on writing a short series of posts that explore various angles of clinical thought. These are talks I have had with medical students, residents, and my patients, to make my thought process as transparent as possible. I have found it to be extremely useful and a worthwhile investment in time.
The Dr. House syndrome
Patients often come to their doctor’s visits with a hidden narrative, a narrative I call the Dr. House approach. This is a narrative the public has largely learned from watching doctors on TV. It is also a perfectly reasonable narrative, and is not wrong as far as it goes. It’s just very narrow and incomplete.
The narrative is this – a patient has an illness (a certain set of symptoms), and that illness is caused by a specific pathological process or disease. The job of the physician is to be smart enough to make the correct specific diagnosis, which will then lead directly to a cure and resolution of the illness.
When this sequence of events does not occur or fails for some reason it is solely due to the fact that the physician simply did not make the proper diagnosis, so they should try again, order more tests, or refer the patient to someone with greater knowledge or more appropriate expertise.
On TV at least, Dr. House always eventually makes the correct diagnosis and saves the day.
Sometimes this is actually how medicine works. In fact doctors love telling each other stories about the complex or difficult case that they eventually diagnosed, especially when that further led to a cure, or at least an effective treatment. Often these stories are formally told as case reports or presented at conferences. These are fascinating stories because they are the exception, rather than the rule, of our day-to-day practice.
Other clinical pathways
While this is one possible pathway that diagnosis and treatment may take, it is not the only legitimate pathway. Here there is much variability by specialty. For most specialties, however, most patients will present with common ailments and diseases (common things are common – it may seem obvious but we do have to constantly remind ourselves of this – in fact this one point is worthy of its own post).
Diagnosing common conditions often is a matter of making a specific diagnosis and treating it. Patients have hypertension, diabetes, multiple sclerosis, etc.
What I am discussing here are those patients with something other than discrete, common, and easy to establish diagnoses. How do we approach such cases?
Ruling out
It is often as important to know what a patient does not have as knowing what they do have. We order some tests to rule out diagnoses that are possible, serious, and/or treatable. (An algorithm as to which diseases to consider and which tests to order can be the topic of a future installment in this series.)
It was surprising to me when I first realized how upset some patients can get when you inform them that they do not have a horrible diagnosis. I actually had one patient cry when I gave them the “good news” that their test was negative. That’s because I had yet to understand the narrative under which they were working. They were thinking diagnosis leads to treatment leads to cure – no diagnosis, not treatment.
I have since learned that I have to explain to patients as I am giving them their results that it is a good thing when certain tests come back negative. They do not want to have the diagnoses I am ruling out.
Further I explain that it is sometimes better to have no diagnosis than to have a bad diagnosis. Patients in whom all the tests come back negative (depending on the clinical situation) often do better than those with a specific diagnosis.
Further still, it is often not necessary to have a specific diagnosis in order to treat an illness. (Again, this is very specialty-specific – surgeons do need a specific diagnosis most of the time, whereas neurologists often have to proceed without one.)
To treat without a diagnosis
To summarize everything above, an alternate narrative to “diagnose, treat, cure” is this: Rule out everything bad, understand the phenomenology as best as you can, then treat symptomatically. This may seem less satisfying, but it is often the preferable situation (again – you do not want to have a horrible diagnosis just to have a diagnosis).
Migraine is an excellent example of this. Patients present with a set of symptoms that we recognize and categorize as a migraine. There is a great deal of variability, however, and many patients may present with an atypical migraine, or migraine-like syndrome. There is no way to objectively make this diagnosis, no positive diagnostic tests.
The diagnosis is partly based on understanding the phenomenology – what kind of process is happening. It is also partly based upon ruling out other things that can cause similar symptoms (like a brain tumor). We then treat based upon a categorical diagnosis – the kind of process that is likely happening. Response to treatment then becomes the best confirmation we can get that the diagnosis is probably correct.
This may all sound very unsatisfying, but this just exposes yet another difference between public and expert narratives. Doctors learn to think in terms of what is likely to happen. How is this patient’s story going to play out? After years of practice you develop a perspective on this, and you realize that it is not a terrible thing when all the tests come back negative. Those patients may do quite well.
Another way to think about the above approach is this: The diagnostic workup is focused on finding active processes and pathology, especially those that are treatable. We then treat what we find, in addition to treating symptoms to improve quality of life. At no point in this process do we need to establish a specific diagnosis.
For example, I see patients with nerve damage (neuropathy). First I establish that the patient’s symptoms are due to nerve damage and characterize that as best possible. Then I look for any active process or anything treatable that might contribute to poor nerve health. I may find that the patient has a low vitamin B12 level. For many patients I still have no idea if the low B12 is causing or contributing to their neuropathy. Except in advanced cases, this is difficult to determine, and I won’t get a nerve biopsy just to confirm this. Instead I will just replace the B12.
Categorical and vague diagnosis
Another important distinction to make is that not all diagnoses are created equal. This too will be the topic of a future post, but for now suffice to say that there are hierarchies of diagnosis, and different types of diagnosis. Not all diagnostic labels refer to a specific and distinctive pathological entity. Sometimes we can determine what type of process is happening (an inflammatory process, for example) but not get more specific than that.
In other words, making a diagnosis is not a black or white, all or nothing proposition.
This approach to some patients has led, in my opinion, to the creation of vague diagnostic categories. This is because we need something to label patients in whom we have ruled out everything specific. These categorical diagnoses can be useful, but also a trap if you treat them as if they were a specific pathological diagnosis.
Fibromyalgia is the poster child of this phenomenon, in my opinion. It is a convenient label for anyone with muscle and connective tissue pain (myofacial pain) that is otherwise undiagnosed. Sometimes clinicians will call such a label a “garbage pail diagnosis” – we throw everything in that we cannot label more specifically.
This can be a useful placeholder, it can be useful to put the diagnosis into its proper category in terms of which kind of process is happening, and it is convenient short hand. In this case fibromyalgia may also be a specific disease, with specific findings, but it is used as a catch-all for any myofacial pain syndrome.
I prefer to use the term “myofacial pain syndrome” because it does not imply more specificity than actually exists. There is a bias, however, toward labels that sound like specific diseases.
Don’t be complacent
All of this is not to suggest that we should be complacent with less than complete knowledge. This is a tricky balance that also benefits from experience. We have to understand the limits of our knowledge and technology, and understand how to approach and treat patients in whom we will not make a specific diagnosis. But this should not lead to complacency – accepting too quickly that we don’t have a diagnosis.
The most important factors in determining how aggressive our posture should be toward making a more specific diagnosis is whether or not we have found specific pathology, and how the patient is doing clinically. If the patient is stable or improving, and/or responding to treatment, it may not be necessary or advantageous to continue ordering more diagnostic tests. If the patient is progressively getting worse, however, this suggests the need for more information.
Also, sometimes tests come back negative and sometimes they come back with abnormal but non-specific findings. For example, an MRI scan of the brain may find a “lesion” (a deliberately vague word) but not show specific features that would allow for a diagnosis of what the lesion is, exactly. We now have possible evidence of a pathological process, and this warrants a more aggressive workup than if the MRI scan came back pristinely normal.
Conclusion
The goal of this series of posts is to help physicians and patients communicate more effectively by exploring and understand the various assumptions and narratives that each bring to the clinic visit. This is an exercise in metacognition – thinking about thinking, in this case thinking about how doctors think.
As you can see, it quickly gets complicated. I have raised many points here, each of which deserve further exploration. The comments are likely to raise points and questions that I can also explore further in future posts.
In approaching this series the most challenging aspect was figuring out how to divide up the various topics. They all are interrelated in a web of decision-making. Hopefully the result was not too disjointed, but I will attempt to keep referring back in future posts and hopefully it will all come together in the end.
Other entries in this series
Part IIPart III

Clinical Decision-Making: Part II
How do doctors make clinical decisions to arrive at a diagnosis? Part 2 of a 3-part series.
Steven Novella on March 13, 2013
Link
This is the second in a brief series of posts about how clinicians think. My purpose here is to elucidate how skeptical principles apply to clinical decision-making, but also as background to provide context to many of the articles we publish here. In this installment I will review the factors that clinicians consider when deciding what tests to order for screening and when conducting a diagnostic workup.
The gunshot approach
Last week I discussed the “Dr. House” approach to medicine, using that particular TV character as an example of how medicine is often portrayed in fiction. Another aspect of the Dr. House image that is very misleading is his approach to diagnosis, which tends to be very linear. He decides what the most likely diagnosis is, then proceeds to either treat that entity or order a confirmatory diagnostic test. When that diagnosis fails, he then proceeds onto diagnosis B. A string of such failures then culminates in a flash of brilliance that allows him to make the actual obscure diagnosis and cure the patient. This approach is optimized for storytelling and drama, but is not how actual clinicians operate.
At the other end of the spectrum is what doctors often refer to as “the gunshot approach” – test for everything in hopes that you hit something. Another derogatory term that doctors throw around is “a fishing expedition,” referring to a diagnostic approach that amounts to hunting around for any possible diagnosis without having a real justification.
A more optimal approach lies somewhere between Dr. House’s serial approach and mindlessly testing for everything. Doctors tend to take a layered approach, using various criteria to decide which tests are worthwhile, which tests have to be done, and which are not justified. Often there is a tiered approach; if round one of diagnostic testing does not yield a positive result, then progressively less likely diagnoses can be pursued. When this approach is persistently negative, then there is the trick of knowing when to stop – when further testing will yield diminishing returns.
Criteria for testing
Here are the various criteria that diagnosticians use to determine which tests should be ordered. Sometimes there are fairly strict algorithms determined by the standard of care, at other times doctors make a judgement about the cumulative impact of all of these factors combined.
Sensitivity and specificity: Sensitivity is the probability that a test will be positive if the patient actually has the diagnosis being tested for. Failure to test positive in someone with the target disease is called a false negative. Specificity is that probability that the patient has the disease if the test comes back positive. Testing positive in someone without the target disease is called a false positive. The more sensitive and specific an available test, the more useful it is.
How likely is the diagnosis? The probability that a diagnosis is present also dramatically affects how useful testing is. If the probability is very low then even a very specific test will be more likely to generate a false positive than a true positive. And of course, testing is more likely to give you an answer if you are looking for a diagnosis that is likely to be present.
What is the morbidity and mortality of the disease? It is more important to diagnose, or rule out, serious illness. There are, in fact, certain entities that we simply cannot afford to miss. Benign and self-limiting diseases, on the other hand, may not be worth diagnosing since they will get better on their own anyway.
How treatable is the disease you are looking for? The golden rule of diagnostic testing is this – how will the results of your test affect your management of the patient. If you don’t know the answer to that question, don’t order the test. You never want to be in a situation where you have an abnormal result and no idea what to do with it. You should have sorted that out before ordering the test.
How invasive, expensive, inconvenient, risky, or painful is the test? These factors get to the risk vs. benefit calculation of ordering a specific test.
Adding all these things together, it is clear that a doctor probably should not order an expensive, painful, highly invasive, and very nonspecific diagnostic test in order to diagnose a rare and benign entity that isn’t treatable anyway. We should order a simple and highly sensitive and specific test for a common, deadly, and curable disease.
These are two ends of the spectrum, and we encounter every possible permutation in between. Sometimes the standard of care demands a certain approach, at other times we are left to our own judgement. For example, in patients over age 50 who present with a new headache it is the standard of care to order a sedimentation rate as a screening test for temporal arteritis. Even though the diagnosis is unlikely (given everyone over 50 presenting with a headache), the test is a simple blood test, and the diagnosis is highly treatable and very severe when not treated, potentially leading to rapid and irreversible blindness. The test is highly sensitive but not very specific, so when it is positive it is usually followed up by a more invasive biopsy, which is highly specific.
On the other hand, when patients over 60 present with dementia the most likely diagnosis is Alzheimer’s disease (AD). We do not, however, perform any testing for AD, because at this time the only useful testing would be a brain biopsy, and this is not justified because it would not affect our management. Instead we order imaging, EEG, and blood tests looking for treatable causes of dementia, even those that are much less likely than AD. If the standard treatable causes are ruled out, then we make the diagnosis of “Alzheimer’s type dementia” and treat that symptomatically. AD is a pathological diagnosis and we cannot make it without tissue, which is not worth getting at this time. There is some benefit to having a tissue diagnosis for family history purposes, but this can be obtained at autopsy with no risk to the patient.
Another way to combine all these factors is to consider the overall risks vs. benefits of several clinical approaches. Making a specific diagnosis with a laboratory test is just one approach. Sometimes it is easier and better to simply treat a probable entity rather than test for it. If the treatment is fairly benign and effective, and the test is less so, sometimes treating without testing is the better approach. Sometimes the time it would take to get the results of the test are simply too long, and treatment decisions have to be made in the meantime.
Screening tests
There are also different clinical contexts for diagnostic testing. A diagnostic test (the context above) is performed on someone who is symptomatic and in whom there is reason to suspect the specific diagnosis. A screening test is performed on a population before they are symptomatic in order to either assess the risk of developing a disease or detect a disease very early in its course when it is more treatable or to prevent morbidity.
The topic of screening has been discussed many times on SBM. The counterintuitive point that often needs to be made is that more screening is not always better. It’s possible for the negative consequences of testing to outweigh the benefits. This threshold is usually determined by how likely it is for a positive screening test to be a true positive vs. a false positive. Even for a test that’s 99% sensitive and 99% specific (which is better than most diagnostic tests), if the prevalence of the disease being screened for is one person in 1000, and you screen 100,000 people, that will result in 999 false positive tests, and 99 true positives (with one false negative). The false positives greatly outnumber the true positives. You then have to consider what the response is to a positive screening test, which may be a more invasive follow-up test, or a treatment that has its own risks. You even have to consider the anxiety and stress produced by the false positive tests, if this is for a serious or stigmatized disease.
Target populations also have to be identified. Screening tests do not necessarily have to involve the general population. They can be targeted at high risk populations, determined by age, sex, family history, or other risk factors.
Conclusion
This discussion of criteria for diagnostic testing relates to Part I of this series in which I discussed the utility of making a diagnosis at all. It is not always necessary to optimally manage a patient. I find that patients, meaning the lay public, often assume that more testing is always better, and that making a highly specific diagnosis is a prerequisite to proper management.
The reality is that diagnostic testing is just another part of the risk vs. benefit calculation at the heart of all clinical decision-making. Often the most difficult decision to make is to decide which tests not to order. It is the general experience of doctors, backed up by published data, that specialists tend to order fewer tests. With greater experience and knowledge in their specific area, they are more likely to perform a targeted workup and avoid the “gunshot approach.”
This can often be difficult to explain to an anxious patient who wants a diagnosis. This can further place a great deal of pressure on the doctor to simply order the test (combined with liability to lawsuits when unlikely outcomes turn out to be the case). These factors combine to drive up the costs of health care, a consequence that is getting increasing attention as these costs continue to rise.
One solution, which does occur but needs to have a much greater place in the practice of medicine, is published guidelines and standards. Doctors can feel more confident in not ordering a test if published guidelines tell them it is not necessary. A published standard of care also effectively shields them from lawsuits – you cannot sue for a bad outcome, only for failing to practice within the standard of care.
Diagnostic guidelines that are well-established and published need to be communicated more thoroughly to doctors in practice, and we can also do a better job at monitoring compliance to such standards.
Other entries in this series
Part IPart III

Clinical Decision-Making Part III
How do doctors make clinical decisions to arrive at a diagnosis? Part 3 of a 3-part series.
Steven Novella on March 20, 2013
Link
In part I of this series I discussed clinical pathways – how clinicians approach problems and the role of diagnosis in this approach. In part II I discussed the thought processes involved in deciding which diagnostic tests are worth ordering.
In this post I will discuss some of the logical fallacies and heuristics that tend to bias and distort clinical reasoning. Many of these cognitive pitfalls apply to patients as well as clinicians.
Pattern recognition and data mining
Science, including the particular manifestation we like to call science-based medicine, is about using objective methods to determine which patterns in the world are really real, vs. those that just seem to be real. The dire need for scientific methodology partly results from the fact that humans have an overwhelming tendency to automatically sift through large amounts of data looking for patterns, and we are very good at detecting patterns, even those that are just random fluctuations in that data.
Many cognitive errors that plague any attempt to investigate the world (including clinical investigations) are some variation on the basic concept of mining large amounts of data in order to find apparent patterns, and then assuming the patterns are real or using confirmation bias to reinforce the perception that the pattern is real.
We have written many times about this concept as it applies to research – making multiple comparisons, sub-group analysis, publication bias, applying various statistical methods, poor meta-analysis, and blatantly cherry picking studies are all ways to pick signals out of the noise. But this happens in the clinical setting as well. In fact it is more of a problem in the clinical setting which is by necessity anecdotal and often cannot benefit from blinding or repetition.
One way to mine data is to order a large number of tests. The normal range of test outcomes is usually determined by taking two standard deviations on the Bell curve of results from a healthy population. This means that 95% of the results will be in this range, and 5% of tests in healthy individuals will be outside the range. Order a battery of 20 tests (like a chem-20 blood test) and on average one will be abnormal by chance alone.
This example is fairly obvious and most clinicians learn about it as a medical student. There are more subtle forms, however. In a patient with a difficult diagnosis it is common for them to go through multiple diagnostic procedures in series, until something abnormal is found. It is then tempting to conclude that the abnormal test is causally related to whatever symptoms were being investigated. It is also possible, however, that the clinician simply kept ordering tests until they got a false positive.
The same process applies to treatment. A clinician (or a patient seeking treatment from multiple clinicians) may try one treatment after another until they finally find one that works – or until their symptoms spontaneously improve in which case they will attribute that improvement to the last treatment they happened to try.
A partial solution to the above pitfalls is to order confirmatory tests to follow up the positive, or in the case of treatment to stop and then restart treatment to see if the beneficial effect goes away and then returns.
A great deal of data mining also occurs in the process of taking a history, or in the patient’s history itself. Patients, for example, will often search their memory for anything that may have caused their symptoms. They generally employ open-ended criteria (trying to think of anything interesting, rather than a specific cause), and underestimate the noise in their everyday lives.
The result is that it is almost always possible to think of something that happened, some exposure, some minor trauma, a stressor, a life change, a new environment – and then assume a causal relationship. Confirmation bias then kicks in to reinforce this belief, and that leads us to the next section.
Mechanisms of confirmation bias
Confirmation bias is a general term that refers to cognitive processes that tend to reinforce beliefs we already hold. This is mostly thought of as remembering hits (confirming information) and forgetting or dismissing misses (contradictory information).
The fallibility of human memory also aids tremendously in confirmation bias. Patients will not only differentially remember information that reinforces their narrative, their memory will become progressively contaminated and distorted to further reinforce it.
One example of this is anchoring. We have a poor memory for how long in the past an event happened. We tend to “telescope”, which means underestimating how long in the past an event was. When a patient says they had their symptom for 1 year, it is likely that it was really 2-3 years.
Patients, however, may also anchor one event in time to another event. This may be accurate and therefore helpful if they are anchoring to a public event that fixes their memory in time. Just as often, however, the anchoring is false, an artifact of their evolving narrative.
For example a patient may have mentally anchored the onset of their headaches to a minor car accident they had, because they have come to believe that the car accident is the ultimate cause of their current symptoms. This is not an unreasonable hypothesis, but the details of the patient’s memory over time will change to fit this story. After a year of telling their story to different doctors they may report that they were perfectly fine and then immediately after the accident all their symptoms began. Meanwhile their medical records may indicate that they were complaining of headaches a year prior to the accident.
That, of course, is the reason that clinicians need to keep obsessive records. They are a remedy to the vagaries of memory. This is also why anecdotal case reports are so unreliable.
Placebo effects also play into confirmation bias. A treatment that addresses a patient’s narrative is more likely to evoke a positive placebo response than one that does not, and then will be taken as confirmation that the narrative is correct.
The representativeness heuristic
There is a general cognitive tendency to estimate probabilities more based upon how well something fits the typical characteristics of a category than the base rate of that category.
The classic experiment presented subjects with a character profile of a college student – nerdy, good at math, likes computers, likes to spend time working alone – and then asked the subjects to estimate the probability that the student was an engineering major. Most people ranked the probability very high, because the profile was representative of a typical engineering student.
However, only 1% of students at the college are engineering students (the base rate), and when this is taken into consideration it is much more likely that the student is not in engineering.
This reasoning also applies to diagnosis. Clinicians tend to be highly confident in a diagnosis when the patient and their symptoms are highly representative of the typical presentation. This is reasonable as far as it goes, but is incomplete and therefore flawed reasoning. You also have to consider the base rate of the disease in question.
This results in what is called the zebra diagnosis. Medical students are famous for this fallacy – coming to the conclusion that a patient has a rare disease because they have some typical features. They have not yet learned through experience that rare things are rare, and common things are common. When you consider the base rate, even a typical presentation of a rare disease may not be the most likely diagnosis. An atypical presentation of a very common disease may be far more likely.
Rather than thinking in terms of representativeness, clinicians are better off thinking in terms of base rates and predictive value. A symptom, for example, may be very typical and therefore representative of a particular illness (for example, fatigue is a typical presentation of chronic fatigue syndrome). But that symptom may also be present in the healthy population or with many other diseases. The presence of that symptom may therefore not be very predictive of the particular diagnosis.
Another way to look at this is that some symptoms are more specific to certain diagnoses, while others are non-specific. Every week I have patients come into my office with a list of symptoms they pulled off the internet, convinced that they have a rare or uncommon deadly disease. They are almost always wrong, because they have fallen for this cognitive error. They don’t have the background knowledge to know which symptoms are predictive and which are not, and they are also not familiar with the base rates of various diseases.
In this example another effect also comes into play – the Forer effect. This refers to the tendency to take a general description and then apply it to oneself, finding examples that confirm the description. This not only applies to horoscopes and psychic readings, but lists of symptoms on the internet.
The representativeness heuristic manifests in many important yet subtle ways in clinical practice. For example, if you are a woman in the emergency room having a heart attack you are less likely to be properly diagnosed and treated than if you are a man with the same presentation. This is because we are biased to think of a middle-aged man as the typical heart attack patient – they are more representative.
The toupee fallacy
You may think you can always tell if someone is wearing a toupee, but the problem with this observation is that you have no idea when you don’t recognize when someone is wearing a toupee. This fallacy applies to diagnosis as well. When you look for a disease you may be likely to find it, but you have no idea how often the disease is present when you don’t look for it.
This is not completely true, as the patient may seek out a second opinion or you may refer to a specialist who does make the diagnosis. (This is why it is critical to give feedback to clinicians who initially missed a diagnosis.) Or the disease may progress and the diagnosis may declare itself.
For many benign, self-limiting, or chronic stable conditions, however, a proper diagnosis may either not be possible with current technology, or may never be made. A clinician can convince themselves, however, of uncanny diagnostic acumen if they only look at the positive outcomes and never look at the data systematically.
Related to this is the congruence bias – the tendency to only test your own hypothesis, and not competing hypotheses. In medicine this manifests when taking a history and ordering tests, which are a way of testing your clinical hypotheses.
If, for example, you suspect that poor sleep is a major contributor to a syndrome, such as headaches, you may ask all your patients with headache about the sleep quality and find that most have poor sleep. This will seem to confirm your hypothesis. However, if you also asked your patients without headaches you may find a similar rate of poor sleep.
Conclusion
Both patients and clinicians are subject to a long list of cognitive biases and errors in thinking that conspire to confirm beliefs about the cause and effective treatment of symptoms and illness. This is why, without the anchoring of adequate scientific methods, and a healthy dose of applied skepticism, practitioners will tend to slowly drift into a world of quackery.
Any treatment can seem to work, and any made up disease may seem to be real, when we rely only upon our naïve reasoning. The pages of SBM are full of dramatic examples.
Other entries in this series
Part IPart II

   Diagnostic Dilemmas
Harriet Hall 
Link
Sometimes diagnosis is straightforward. If a woman has missed several periods and has a big belly with a fetal heartbeat, it’s pretty easy to diagnose pregnancy. But most of the time diagnosis is much more difficult. Alzheimer’s can’t be diagnosed for sure until the patient dies and you do an autopsy. If only we had one of those Star Trek gadgets to point at our patients and give us a quick and accurate answer! Alas! We are far from perfect. All too often, we really have no idea what’s causing a patient’s symptoms. We do a complete workup and still don’t know. What then?
We all know people who have symptoms that a series of doctors have failed to diagnose, who continue to doctor-shop, hoping to find that one doctor somewhere who will find something the others have missed. Occasionally they do; but far more often these people spend a great deal of time and money chasing a will-o’-the-wisp. Sometimes as they are searching, the illness gradually runs its course and goes away. When this happens, whatever they tried last gets the undeserved credit for the “cure.” Sometimes the symptoms persist and these searches consume their life, encourage unhealthy self-absorption, and permanently ensconce them in the “sick” role.
One of the attractions of alternative medicine is that it offers far more certainty than scientific medicine. If your scientific doctor can’t see anything on x-rays, your chiropractor can. He’ll tell you he knows exactly what’s wrong: a subluxation that he can fix. Sherry Rogers will tell you all illness is due to toxins accumulating in your cells and you must “detoxify or die.” Hulda Clark will tell you it’s all parasites that she can eliminate with her magic zapper. Robert Young says the cause of all disease is acidosis. They all have confident, precise answers. Wrong ones.
The One Cause of All Disease?
It’s really easy to figure out what’s causing a patient’s symptoms if you believe there is one simple cause for all disease. While I was writing this I got sidetracked and searched the Internet for “the one cause of all disease.” I found a lot of them, including:
Toxemia
Subluxations
Oxygen deficiency
Psora
“‘““Fearful, tight and negative minds”
Obstruction of ch’i along the meridians
Refined sugar
“Fault of awareness”
Grains in the diet
False beliefs and fears
“Imbalance”
Ama due to aggravated doshas
Stress
Anger
Modern medicine
Some morbid agent, producing irritation and inflammation
Arrogance
A “non-perceivable but very real attachment to the material aspect of creation”
Inadequate nutrition
A congested colon
“All disease is a learned experience which we can un-learn.”
“All illness is in our minds,” and we can cure it with faith in God, meditation, or whatever.
Spiritual vital force and its dynamic derangement
Holding on to energy within the physical, emotional and spiritual bodies that is not in harmony with us
Impairment of movement of the bones of the skull
Bad health habits
Nerves too tense or too slack
God
Lack of life
Witchcraft
Miasms
Poor sanitation
A shock experience that catches us completely off guard
Cellular memories
An excess or insufficiency of nervous tension
Poor digestion
Weak digestive fire
Exogenous toxins
Morbid matter
An unbalanced life style
Disharmony in the equilibrium of Yin and Yang
A breakdown of the immune system
A weak “immine” system
Malnutrition
Free radicals
An imbalance of electrons in the cellular atoms
Emotion
Sin
Food abuse
Allergies
Ignorance of reality
Dis-ease on any level (physical, emotional, mental, soul or spiritual) is incorrect vibratory rate(s), patterns which are not appropriate, or blocked energy pathway(s) within or between the various levels of existence
Repletion
The blood
Morbid humors
Poisonous chemicals
Emotional trauma
“Allurement” of the mind by sense objects and its “willfulness” in gratifying these desires
Toxic metals
Cold
Blocked nerves
Our inability to adapt
Overeating
Poverty
Food acidity
Violation of natural law
And my favorite: “the United KKK States of America is the root cause of all disease…”
I’m sure there must be more, but I ran out of steam. You might ask how there can be more than one “One Cause.” Proponents of these different One Causes don’t usually fight each other. They are more likely to say that “your truth may not be the same as my truth, and that’s OK,” and “there are many realities.” But I’ll stick to the reality that science deals with, thank you very much.
If You Don’t Think There Is Only One Cause
So if all you want is a definite answer, there are plenty of (wrong) definite answers to choose from. But what if you think there might be more than one cause for all disease and you want to stick to scientific medicine?
What do symptoms mean? Sometimes they indicate a serious condition requiring urgent treatment to save life or limb. Sometimes they indicate an illness that medical science hasn’t yet figured out how to diagnose or treat. Sometimes they indicate a benign condition that will resolve on its own. Sometimes they may be due to hyper-awareness of normal body functions, or to extremes of the normal range of physiologic processes. Sometimes they represent somatization related to depression. Sometimes they are signs of malingering. Sometimes they are an excuse to talk to someone and get attention. Sometimes they are a surrogate for something else.
When I was an Air Force doctor in Spain, a sergeant’s wife came in with a whole laundry list of complaints: headaches, backaches, fatigue, you name it. I happened to know that she was a Spanish woman who had been living in the States but now she and her two small children were back living with her parents in a little Spanish village while her husband was on an unaccompanied tour of duty where she couldn’t go with him. I asked her one question: “Isn’t it hard for you to be back under your parents’ roof after living on your own in the States all these years?” She burst into tears and vented her many frustrations. We talked about how she could better cope with her difficult situation. I saw her for several followup visits. She never even mentioned her physical symptoms again and I didn’t need to do any kind of diagnostic workup.
Diagnosis Can Be Deadly
How can we decide when a symptom deserves an extensive diagnostic workup and when it doesn’t? Our strong desire to know can be counterproductive.
When I was a medical student, a patient had liver disease and was getting better. Getting better wasn’t good enough for his doctors: they wanted to know what he was getting better from. They did a liver biopsy. His disease had impaired his clotting mechanisms; he bled, he developed DIC (disseminated intravascular coagulation) and he died. Ironically, even the autopsy failed to establish the elusive diagnosis.
I heard about an Air Force general who was killed by his routine physical. He mentioned that he had frequent headaches. They sounded like typical garden-variety tension headaches, but since he was a VIP and they wanted to be sure not to miss anything, they ordered a brain scan. This was in the days before CTs and MRIs, and the brain scan they did was a fairly blunt instrument. The radiologist thought it was probably normal, but he couldn’t completely rule out the possibility of a subtle abnormality, so they proceeded to do an angiogram, injecting contrast material into his arteries, and the procedure caused a stroke and he died.
My husband’s father was killed by a routine chest x-ray, back in the days when we did annual chest x-rays on everyone. They saw a spot on his lung, thought it might be cancer, operated, found out it was benign, and then one post-operative complication led to another until he died. He didn’t need the chest x-ray in the first place, and he wouldn’t have gotten it today.
Tests Are Imperfect
Doing too many tests only increases the probability of finding a result that is abnormal but meaningless. To figure out what normal laboratory values should be, they test a lot of healthy people to create a bell curve of normal distribution. Then they cut off the ends of the curve and call the middle 95% normal. Some normal people will necessarily fall outside the “normal” lab values. If you do a panel of 20 blood tests on a perfectly healthy person, one is likely to come out positive just by chance.
A good rule of thumb is never to believe one lab test, especially if you are going to base a diagnosis on it or start treatment. My mother’s doctor tested her blood sugar and got a sky-high reading. He immediately diagnosed diabetes and prescribed medication, a diet, and home glucose monitoring. I told her not to take the medication yet, and we tested her blood glucose at home repeatedly over three weeks. Every single reading was well within normal limits! She was now labeled as a diabetic on her medical records, but she clearly didn’t have diabetes. Her doctor was skeptical of the home readings, and he ordered a Hgb A1C to double-check. It was perfectly normal.
Lab errors can occur for many reasons. Specimens are inadvertently switched or contaminated, machines malfunction, results are mis-copied. When you get an unexpected abnormal reading, it’s always prudent to repeat it.
And any test is likely to vary somewhat from day to day. If you get a cholesterol of 150 today, you might get 140 tomorrow and 160 the next day. Varying results are partly due to natural day-to-day and hour-to-hour fluctuations in physiologic functions. They are also partly due to the imperfect accuracy of the testing method. There is a margin of error for every test: I had a patient who would agonize over a rise in cholesterol of 2 points, and I tried in vain to tell her that it didn’t mean her cholesterol was up – it was within the error range of the test.
Even doctors tend to forget how imprecise our diagnostic methods really are. When a patient has a sore throat, a positive culture for Strep may not mean that Strep is the cause of the sore throat. The patient might have a viral sore throat and only be a Strep carrier. Herniated disks are seen on x-rays of many asymptomatic patients, so when someone with a backache has positive x-rays it could be just an incidental finding and his back pain might have another cause entirely.
When we do any test, we need to consider the prior probability of the disease we’re testing for. Every test has false positives. The smaller the likelihood of the disease, the less likely a positive test will mean anything.
If you have a positive mammogram, how likely is it that you actually have breast cancer? In a study in Germany, they asked doctors for their estimates. The most frequent response was 90%. Even gynecologists (who ought to have better knowledge than, say, a dermatologist) guessed 90%. In fact, only 10% of women with a positive mammogram have breast cancer on biopsy.
They taught us in medical school that in making a diagnosis, the patient’s history contributes 70%, the physical exam 20%, and the lab tests, x-rays and other procedures only contribute 10%.
Everyone wants a diagnosis. One patient was delighted that her new doctor finally figured out that she suffered from cephalalgia; all the others doctors had passed it off as a headache! (For readers who may not know, “cephalalgia” is just another word for headache, derived from the Greek so it sounds more important.)
But sometimes we just can’t make a diagnosis. For FUOs (fevers of unknown origin), a thorough diagnostic workup fails to account for 5-15%. Fortunately, these generally have a benign long-term course, especially if there are no other signs of serious underlying disease.
What Can We Do?
The general public (and some newly minted doctors) think that if you just look hard enough, you ought to be able to identify the cause of any symptom. That might be true in principle, but not in practice. A good doctor develops some judgment about what tests to do and when to stop.
I used to tell patients that the tests hadn’t picked up anything abnormal, and that I didn’t know why they were having symptoms, but I was reasonably certain that we had ruled out all the serious, potentially life-threatening causes. There were always more tests we could do, but the chance of finding anything was diminishingly small, and there was a significant chance of getting false positive results and going on a wild goose chase, possibly even including dangerous invasive tests or surgery. I would suggest that for the time being we stop worrying about “why” and simply try to deal with the situation and control the symptoms. Three things could happen:
1.The symptoms could go away, in which case it really wouldn’t matter what had caused them.
2.They could stay the same and we could try different ways of coping with them.
3.They could get worse (or new symptoms could develop), in which case we could re-evaluate and consider further testing.
Most patients seemed to understand this and accept it.
It’s scary to take that course, because the lawyers are watching us, and if we ever miss anything or delay a diagnosis, we’ll see them in court. But if we really want to do what’s best for our patients, we have to know when to stop. We have to gain our patients’ confidence so they will trust our judgment and not run off on a doctor-shopping expedition or fall into the hands of quacks. We have to be willing to say we don’t know. We have to educate our patients to understand that medicine deals with uncertainties. Most of all, we have to convince our patients that we care, and that we can keep trying to relieve their suffering even when we can’t pinpoint a cause.

We Should Abandon the Concept of “Alternative Medicine”
Steven Novella on August 12, 2015
Link
In a recent editorial for the New York Times, Aaron E. Carroll argues, “Labels Like ‘Alternative Medicine’ Don’t Matter. The Science Does.” I agree with this headline thesis, but the details of his argument ironically show the harm that the so-called CAM (complementary and alternative medicine) movement has done.
Carroll starts out well, essentially pointing out that the division between “conventional” and “alternative” medicine, and the division between “Western” and “Eastern” medicine are false dichotomies. Despite this strong start, he muddles his way through the rest of his editorial.
The primary error he commits is to swing from a false dichotomy to a false equivalency, essentially saying that there is no difference between conventional and alternative practice or practitioners. In order to support this contention, however, he has to distort the facts beyond recognition.
In other words, Carroll commits the less-well-known false continuum logical fallacy. Let me explain.
The real differences between SBM and CAM
I agree, and it is a central thesis of SBM, that the category of CAM (whether you call it holistic, alternative, complementary, or integrative medicine) is a manufactured and misleading one. In this way it is false, but not in the sense that Carroll takes it, meaning that it doesn’t exist.
Rather, medical treatments, claims, diagnostic procedures, and even concepts of disease and health exist on a scientific spectrum, from rock solid at one end, to completely disproven and utterly absurd at the other.
Conventional medicine attempts to be science-based, and has an infrastructure designed to maintain high scientific and professional quality. But still, this is a challenging and complicated goal. Acceptable components of mainstream medicine therefore range down the spectrum of science to the gray zone, and different people and institutions will draw the line of acceptability in different places.
The philosophy and goal of SBM is to examine this very question – how do we best use logic and evidence to determine where a medical claim is on that spectrum, and how do we know where to draw the line?
CAM exists in that it is a definable approach to this issue that is distinct from mainstream medicine. We explore this at length in the pages of SBM as well. I think we have clearly established here that it is fair to say CAM practitioners in general will use treatments anywhere along the scientific spectrum with little or even no regard for logic or evidence. Further, they are actively trying to weaken and muddy the scientific and academic standards of medicine and the regulations that maintain them (with an unfortunate degree of success).
While CAM may not explicitly be a set of practices, it is a definable set of behaviors that are decidedly unscientific. The CAM approach consistently leads to the same set of practices, like homeopathy, herbalism, energy medicine, and acupuncture, that are not scientific.
The false equivalency
Carroll, while he seems to sort-of get it when it comes to the false dichotomies promoted by advocates of CAM, misses these very real distinctions.
He correctly points out that:
People often think of Eastern or alternative medicine as more “natural.” Many feel that Western medicine is built around technology and products produced in a lab. They’re not entirely wrong. Many of the gains that have been made in traditional medicine have been the result of innovation in laboratories.
But that doesn’t mean that everything doctors are taught in medical school involves a drug or device. I talk to patients all the time about diet and exercise.
I agree – “natural” is a false distinction. What is “natural” about claiming to manipulate non-existent energy, or sticking needles into people? He also correctly points out that nutrition, exercise, physical therapy, and relaxation are all science-based components of mainstream medicine, which is not all drugs and surgery.
Carroll, however, goes off the rails in two important ways. First, he struggles to find CAM modalities that have been validated by science. His examples are all terrible, and in fact demonstrate how scientifically vacuous CAM really is. Even someone dedicated to finding the best examples of science-validated CAM can’t really do it.
He argues that acupuncture is now evidence-based, referring to the infamous Vickers review and concluding:
They found that not only did acupuncture work better than no-acupuncture control groups, but there were also significant differences between acupuncture and sham acupuncture. This suggests that not all of the benefits are placebo effects.
This only demonstrates how far the standards have fallen, especially when considering a CAM treatment. If acupuncture were a drug, with the exact same level of evidence, it would be attacked as a Big Pharma conspiracy.
David Colquhoun and I did our own review of acupuncture, and point out that the most rigorous studies are all negative, no effects are adequately replicated, and the so-called positive effects from meta-analyses such as Vickers are so razor thin that they are clinically insignificant (meaning explainable by even subtle bias or methodological flaws). The best conclusion we can make from thousands of studies is that acupuncture points don’t exist, that it does not matter where (or even if) you stick the needles. In other words, the only scientifically valid and consistent conclusion based on thousands of studies is that acupuncture is little more than an elaborate placebo.
Carroll also mentions several herbal treatments. Herbs, as we have pointed out numerous times, are drugs. A treatment effect is therefore plausible. Herbs are the low hanging fruit of CAM when it comes to scientific plausibility. And yet here too his examples are mostly dubious, with the evidence being weak and contradictory. Orac addresses these specific claims in his own post, so I won’t repeat them here. Carroll also glosses over the many fatal problems with the supplement industry that make it nearly impossible to apply herbal products in medical practice.
The second way in which Carroll goes off the rails is to completely, in my opinion, mischaracterize how mainstream vs CAM practitioners respond to evidence. He makes an explicitly false equivalency argument. After reviewing the evidence that homeopathy does not work, he states:
My friends who believe in homeopathy don’t really care.Those who favor conventional medicine, though, can be just as blinded. Too often, when confronted with evidence that advanced technology might not be providing benefits, the medical community refuses to change its behavior.
Saying that mainstream doctors are “just as blinded” to evidence as homeopaths might be considered defamatory. This statement is so divorced from reality it is hard to imagine how Carroll came to this conclusion. It seems to me he had his thesis – that of the false dichotomy – which he misinterpreted as meaning there must also be an equivalency, and then motivated reasoning kicked in.
Keep in mind, much of what we do here at SBM is advocate for higher standards of evidence within mainstream medicine, and we are quick to criticize the slow adoption of evidence-based standards. In the end, however, mainstream medicine listens to the science. It is slower and messier than optimal, by far, but the philosophy and culture is ultimately science-based.
Homeopathy, on the other hand, is completely divorced from science, and is in fact completely pseudoscientific. Their basic principles violate everything we known about physics, chemistry, and biology. They have stuck to their pre-scientific guns despite two centuries of scientific advance. Their claims are indistinguishable from magic formulas and witchcraft, and no amount of evidence of lack of efficacy will dissuade them.
This creates the sense that while Carroll may be familiar with the shortcomings of mainstream medicine and with science in general, he is largely naïve to the realities of CAM. We find that this is generally true outside of SBM, part of which is specific expertise in medical pseudoscience.
This is true more broadly as well. Scientists do not necessarily possess specific knowledge or expertise in the nature and details of pseudoscience. Such expertise is the purview of scientific skepticism, of which SBM is definitely a part. Carroll’s editorial is a dramatic demonstration of the pitfalls of scientists lacking expertise in pseudoscience while commenting on it.
Conclusion
A thorough and fair assessment of CAM leads one, in my opinion, to the ineluctable conclusion that CAM as a category is counterproductive. In practice it is a marketing term, the purpose of which is to lend false legitimacy to medical practices that were once considered fraud and pseudoscience. It is a banner beneath which proponents seek to weaken the scientific standards of medicine, foment confusion as to the nature of medical practice, treatments, and evidence, to infiltrate academia, and to lobby for weaker regulations.
In short CAM is a scam. It should be abandoned, and the medical profession should re-dedicate itself to an uncompromised standard of high quality science.

Selective pressures on alternative medicine
David Gorski on November 24, 2014
Link

(Editor’s note: I was away at Skepticon over the weekend, where I gave a talk entitled “The Central Dogma of Alternative Medicine”. (When the talk’s up on YouTube, I’ll provide a link, of course.) Because of all the fun and travel delays I didn’t get a chance to turn my slides and notes into a blog post yet. Also, I’m on vacation this week. However, this gives me the opportunity to resurrect a blog post from 2007 on my not-so-super-secret other blog, because I think the concept is interesting. I even use it in a slide that shows up in many of my talks (above). I’ve updated dead links and added some text to include relevant links to posts written since. Enjoy, and I’ll definitely be back next week with original material, if not sooner, given that there are others here who might have the temerity to take part or all of this week off.)
I wish I had thought of this one, but I didn’t. However, I never let a little thing like not having thought of an idea first to stop me from discussing it (even if Steve Novella’s also discussed it), and this particular idea is definitely worth expanding upon because (1) it’s interesting and (2) it combines two of my interests, alternative medicine and evolution. I agree with parts of the idea, but it’s not without its shortcomings. Indeed, I’d very much welcome any of the evolutionary biologists who read this blog to chime in with their own ideas.
A colleague of mine, Martin Rundkvist over at Aardvarchaeology, has proposed a rather fascinating idea regarding the evolution of alternative medicine in which he argues that alternative medicine evolves according to certain selective pressures. As you may or may not know, evolution is not just for biology, but has been proposed as a mechanism in cultural memes, for example. Since alternative medicine is a cultural phenomenon, it is not unreasonable to look at such non-evidence-based medicine and hypothesize what might be the selective pressures that shape its popularity and evolution. After all, if we’re going to discourage the use of non-evidence-based medicine or even quackery, it’s helpful to understand it. We already know that alt-med terminology has evolved considerably into the current preferred term, “integrative medicine.” (See also the image above and my blog posts on this evolution here and here.)
Martin primarily considers what the selective pressures are on various alternative medicine modalities and comes to a startling conclusion: Namely, that the selective pressure on such modalities is primarily to select for ineffective treatments. He bases this on two primary forms of negative selection. First, he hypothesizes, there will be selective pressure against modalities that cause obvious harm. According to this concept, such modalities will tend to be eventually recognized as harmful and shied away from by alternative medical practitioners due to fear of lawsuits and government regulations. The second form of selective pressure will come from conventional medicine. In essence, alternative medical therapies that can be shown to have a reasonable degree of efficacy will risk being co-opted by us “conventional” practitioners of evidence-based medicine and thus taken out of the armamentarium of alternative practitioners, whose setting themselves apart from mainstream medicine is very important to their livelihood. This leads Martin to observe that homeopathy is the ultimate CAM therapy:
So, there is evolutionary pressure on alternative therapies to achieve near-zero effect. This is why homeopathy is still around: its main method being the administration to patients of small amounts of clean water, it’s uniquely suited to surviving indefinitely in the alternative-therapy biotope. Homeopathic remedies can neither harm nor benefit patients.
This is a fascinating and lucid insight. Clearly it has some merit. However, it is incomplete. The reason, I would argue, is that the negative selective pressures Martin identified are almost certainly not as potent as he thinks they are, as evidenced by how rare it is for an alternative medical therapy to actually go “extinct.” Indeed, I would argue that selection against harmful or potentially harmful remedies is actually fairly weak and perhaps even nonexistent. After all, black salve is still around after many decades, if not hundreds of years, and it can produce some truly horrifying complications (not for the squeamish). Even though the FDA banned importation of black salve products and they can be demonstrably harmful, they are still around and show no sign of disappearing. Another example is Laetrile. Multiple well-designed clinical trials demonstrated that Laetrile is ineffective against cancer, and it has the well-known potential complication of cyanide toxicity. It, too, shows no signs of disappearing. Of course, perhaps the most popular ineffective CAM therapy that has potentially deadly complications is chelation therapy, which remains widely used among CAM practitioners to treat cardiovascular disease and autism, despite of the extreme biological implausibility of the argument that it should work for either condition and despite there being no good evidence that it does. Indeed, back in 2006 there was even a well-publicized case of an autistic boy who died as a result of chelation-induced hypocalcemia causing a fatal cardiac arrhythmia.
No, there are lots of potentially-harmful CAM modalities out there that show no signs of going away.
Let’s look at the flip side of the negative selection, co-optation of “effective” alternative therapies into mainstream medicine. Once again, this is probably a weaker negative selective force than it might seem. Herbal medicines, for example, are probably the most common of the CAM-type modalities to show some evidence of efficacy in randomized clinical trials. This is mainly because they are drugs. Impure and dirty drugs with widely-varying levels of active ingredient from lot to lot, but drugs nonetheless. The problem for the co-optation of these drugs by conventional medicine is that practitioners of scientific medicine do not like unpredictability in their drugs. They like drugs with a predictable effect; herbal medicines “in the raw,” so to speak, do not fit the bill, particularly when pure pharmaceutical alternatives that lack the contamination and unpredictability of herbs exist. Even if conventional medicine co-opts an herb, for example, it is usually in the form of the active ingredient purified from that herb. For example, if you have breast cancer, you could try to chew on the bark of the Pacific Yew tree for its anticancer properties, but you’d be a whole lot more likely to do better if you took pure Taxol derived from that bark – and took it intravenously. The example of Taxol also suggests that once conventional medicine co-opts an herbal or plant-based remedy, it usually does not supplant the original alternative therapy. After all, all of the “natural goodness” has been extracted from it during the purification of the active ingredient! CAM mavens would often rather take the raw herb or the herb chopped up and compressed into an herbal pill because it’s more “natural.”
As for other non-herbal CAM therapies, even when they’re co-opted by modern medicine (although it’s often arguable whether conventional medicine or CAM did the co-opting), often an “alternative” version remains. The scientific version will be stripped of all the woo, while the “alternative” version will retain it. Think massage therapy and perhaps even chiropractic, which, as I’ve said before, stripped of its woo is nothing more than physical therapy with delusions of grandeur in the form of claims of being able to cure all manner of illnesses that have nothing to do with the spine or the musculoskeletal system.
Finally, there is one last aspect of Martin’s concept that argues against it. Martin states:
Evidence-based medicine, alternative medicine and weaponry change through time because of selection pressure. This means that they evolve and produce a fossil record of discontinued methods and therapies.
Here’s the problem: There actually is no “fossil record” of discontinued CAM methods and therapies. The reason is simple: CAM does not abandon its methods, regardless of evidence and, to a large degree, regardless of harm. Yes, individual treatment modalities may wax and wane in popularity, but they never go away completely. They never go extinct. Think about it a bit. Can you think of a single “alternative medicine” treatment modality that’s ever been completely abandoned because it either doesn’t work, is too harmful, or has been co-opted by conventional medicine? I can’t. CAM is, in the words of James Randi, an “unsinkable rubber duck.” It just won’t disappear. Martin is quite correct that homeopathy, for example, has persisted for 200 years despite no evidence for its efficacy. Aryuvedic medicine has persisted at least a couple of millennia, despite a similar lack of evidence. Ditto most of traditional Chinese medicine, whose real history has been conveniently retconned over the last several decades, making it more popular even outside of China than it’s ever been. Never mind that these systems were developed in a time when very little was known about how the body actually works and are infused with spiritual and religious beliefs. They are still used by many millions, if not billions, of people worldwide. They have left no “fossils.” Of course, as in evolution in biology, this selection, applied over long periods of time, may ultimately eliminate such modalities, but if I were somehow able to call the Doctor to give me a ride in his TARDIS a couple of hundred years in the future, I bet that virtually all of these CAM modalities would still be in use. Part of the reason, I suspect, is that, as Martin pointed out, most CAM modalities do little; there is usually no CAM modality that can supplant existing modalities.
In any discussion of the evolution of CAM, I would be remiss not to look at its primary competition for resources (i.e., patients) in the ecosystem of medicine, namely scientific, evidence-based medicine. EBM has been hugely successful in many areas. Indeed, it can be said to have driven back CAM to a much smaller “ecological” niche than it once occupied. These days, relatively few people rely on CAM modalities when faced with a truly life-threatening illness, such as cancer. The Katie Werneckes, Abraham Cherrixes, and the Chad Jessops of the world (if the latter even had cancer), who treat life threatening cancers with high dose vitamin C, the Hoxsey concoction, or nasty, burning goo like the infamous “black salve,” respectively, are pretty uncommon. The main ecological niches for CAM these days have contracted to two areas. First are “diseases of living.” In other words, CAM has been for the most part relegated to the treatment of what are generally vague complaints that are not exactly diseases or to self-limited conditions. Indeed, one could argue that the strongest positive selective pressure for CAM modalities is how well each one gives the appearance of doing something therapeutic for such conditions, whether it actually does anything or not. In other words, how good of a placebo is it? Or is its timing or method of administration optimally adapted to correlate with the patient’s improvement anyway, allowing the confusion of correlation with causation? The better the adaptation, the more likely a CAM modality will thrive and expand.
The other remaining ecological niche for CAM, I would argue, is in serious diseases for which conventional medicine does not have much to offer. These conditions include diseases such as terminal cancers that have passed beyond our ability to treat them, as well as any manner of chronic diseases for which conventional medicine does not have a cure, such as Parkinson’s disease, chronic pain syndromes, multiple sclerosis, etc. Conventional medicine can treat and often palliate such conditions, but it cannot cure them. In this latter niche, I would argue that the primary positive selective pressure would be how well the CAM modality can inspire belief in its practitioners and hope in its users. The two are related, of course; the more the practitioner believes in the modality the more he or she can sell the patient on it.
Of course, applying evolutionary principles to CAM only goes so far. It’s a highly complex situation, and there are a number of positive and negative selection pressures that one could postulate. Certainly, the marketplace and how much of a feel-good aspect there is to CAM therapies are important. Finally, no doubt, like evolution, there are aspects to CAM proliferation that do not depend upon selection, a CAM equivalent of genetic drift, for example. Indeed, there was a fascinating paper five years ago about why quackery persists which argues for a similar hypothesis of selection for more ineffective therapies but also added a twist: That even in self-medicating or use of alternative medicine effective remedies can be lost due to stochasticity, in other words, due to random chance. Indeed, the authors point out that most highly efficacious innovations would be predicted to be lost due to stochasticity. Indeed, my reading of this study would suggest that one reason why highly effective treatments actually do persist in our society is because scientists, physicians, and science-based medicine validate what treatments are efficacious, retaining the treatments that are and trying (but not always succeeding) to discard the ones that aren’t. For people self-medicating, such a mechanism is not operative.
Since were using evolutionary principles now, one might also look at CAM this way with respect to its “competitors.” Perhaps CAM is adopting an evolutionary strategy not unlike that of dogs or cats. In other words, it was subsuming itself to its more successful competitor, conventional medicine, in much the same way that ancient wolves were domesticated by humans and ultimately started to speciate into dogs. The problem with that analogy, of course, is that both human and dog gain benefits from their relationship. Humans gain companionship and work from dogs; dogs gain protection and a reliable source of food and shelter. It could be argued which species gains more, human or dog (probably humans early on and dogs now), but there is little doubt that both species benefit. So, in the relationship between CAM and conventional, who benefits?
The answer is obvious: CAM. Scientific medicine does not need CAM, but these days CAM appears to need scientific medicine. Indeed, the very name CAM was adopted to allow alternative medicine to seem more palatable to practitioners of conventional medicine and ease its ability to insinuate itself into academic medicine, which, as I’ve documented extensively, it is having increasing success in doing so, even to the point of finding its way into the curriculum of various medical schools.
What this tells me is that we’re looking at a “speciation” event in alternative medicine. There is a strain of alternative medicine that fits in with the whole movement towards CAM in medical schools, and there is a strain of alternative medicine that does not. Now here’s where Martin’s idea of selective pressures favoring placebos in alternative medicine comes in. The strain of alternative medicine that either excessively credulous or cynical academicians embrace falls under exactly the sort of selection pressure that Martin discussed that favors minimal effects. The reason is simple. The two most common justifications used for including CAM in academic medical centers are (1) the patients want it and (2) the perception among academic physicians that it won’t do any harm anyway and seems to make patients feel better. Consequently, the most common varieties of CAM in such settings are massage therapy, meditation, acupuncture, yoga, and nutrition-based therapies. Oddly enough, arguing against Martin’s idea is that seldom will you see homeopathy in academic medical centers, at least in the U.S. I speculate that that is because in such settings, there is also a negative selective pressure against extreme scientific implausibility – at least upon the initial incursion. On the other hand, perhaps it’s just an odd quirk of history, culture, or whatever that makes use of homeopathy less common in the US. After all, reiki is just as ridiculous as homeopathy, if not more so (my talk this weekend prominently featured reiki, for instance), but it’s everywhere these days in academic and community medical centers.
So what term best describes the relationship between CAM and academic medicine? Certainly CAM advocates would argue that it’s a symbiotic relationship. I would counter that it’s a parasitic relationship, and here’s why. Parasitism is defined as a relationship where one organism benefits and the other is harmed. CAM is indeed parasitic. It benefits from its association with academic scientific medicine by obtaining a level of plausibility and respectability that it could never obtain on its own, while it arguably harms academic medicine in the process. True evidence-based medicine is what academic medical centers are ostensibly built to promote, applying the scientific method to medical therapies in order to find more effective treatments. What academic medical centers should be doing is to educate the new generation of physicians in the scientific method, to better prepare them to be able to evaluate claims for treatment, whether they come from conventional medicine or elsewhere. This is how medicine has advanced so rapidly over the last 60 years. Blurring the line between science and non-science, evidence-based medicine and woo, through the enthusiastic promotion of CAM in medical school curricula, harms that endeavor, both by degrading the ability of physicians to think critically (thus preparing them to accept even more implausible treatments) and by wasting money and resources that could be better used to study more promising science-based modalities to study obviously highly-implausible gobbledygook before there’s any good evidence that it does anything at all beyond the placebo effect. It would be one thing if CAM were being studied from a truly scientific perspective. I don’t object to that; indeed, I encourage it. A true scientific examination of the vast majority of CAM will likely find it useless, while a few gems might be pulled out of the dirt. Unfortunately, though, as I’ve mentioned before, that’s not what usually happens. What almost invariably happens is that CAM is used as a marketing tool.
Although I often disagreed with him (mainly about politics) Panda Bear, MD (whose blog seems to have been hacked and has become a spam site) gets it right in describing this parasitic relationship:
Suppose I were to actually build a house. Along with a foundation it would require framing of the walls and floors, siding, wiring, glazing, plumbing and a dozen other skilled trades coordinating their efforts. The practioners of Complementary and Alternative Medicine would be like your Aunt Mildred telling you how to hang the toilet paper in the finished bathrooms and then trying to claim credit as an essential part in the construction. Complementary and alternative medicine only exists because real medicine does all of the heavy lifting leaving a risk-free enviroment [sic] in which it may ply its patent remedies. At best it’s an afterthought, something that legitimate hospitals add to their services to attract the kook money. At worst it’s a cynical ploy to fleece a little extra from the desperate, many of whom are dying and will gladly pay for another straw to grasp. In no way is it an essential part of medical therapy except that it provides entertainment to the patients and their families while medicine and nature run their courses.
He’s not quite right, at least now. These days, CAM is like Aunt Mildred claiming that the house was built by magic.
Conventional medicine fights the real battles and faces the real danger of failure, while increasingly CAM attaches itself to conventional medicine, much as the parasitic roundworm Ascaris lumbricoide finds its way into the small intestine of its host. CAM benefits from its association, but its host, conventional medicine, most definitely does not. At best it is not harmed; at worst, grave harm to scientific medicine becomes possible.
All this speculation leads me to believe that the form of CAM that increasingly thrives in academic medical centers is indeed developing into a new “species” of woo, so to speak. It’s wraps itself in scientific-sounding terminology and, for the most part, discards the more outrageously silly religious and supernatural elements that it can, all in order to become seemingly inoffensive enough that academic physicians, although they may not approve of it, remain insufficiently sufficiently alarmed by it to rise up and purge the system of this parasite. Meanwhile, the parasite grows in number and strength, continually weakening the body of academic medicine the longer it stays. Eventually, like the roundworm, it spreads its eggs where more and more academic medical centers can pick them up until the species is so entrenched and self-perpetuating that it can’t be dislodged without resulting in severe injury or death to the host.

“Quackery: A $10 Billion Scandal”
Jann Bellamy on October 30, 2014
Link

Who would you guess authored a 250-page report which begins with this Preface?
This report marks the culmination of an intensive four-year review of quackery and its impact on the elderly. . . As this report details, quackery has traveled far from the day of the pitchman and covered wagon to emerge as big business. Those who orchestrate and profit from the sale and promotion of these useless and often harmful “health” products are no longer quaint and comical figures. They are well organized, sophisticated and persistent. [We estimate] the cost of quackery – the promotion and sale of useless remedies promising relief from chronic and critical health conditions – exceeds $10 billion a year. The costs of quackery in human terms, measured in disillusion, pain, relief forsaken or postponed because of reliance on unproven methods, is more difficult to measure, but nonetheless real. All too frequently, the purchaser has paid with his life. While the impact of quackery on our lives has been increasing and growing in sophistication, public and private efforts designed to address and control this problem have diminished, been redirected or disbanded.
The author defines quackery as:
The promotion of medical remedies known to be false or which are unproven for a profit, usually by means of false representations that they will “cure” or aid in the cure of various diseases and problems. Quackery is pseudoscience at its best.
And offers the same observation we’ve made here at SBM that:
one of the most appalling elements of quackery [is that it amounts to] uncontrolled tests on human guinea pigs . . . Humanity as well as science demand that new remedies be tested and proven safe and effective before wide dissemination.
The author goes so far as to suggest that this:
is reminiscent in its callous disregard for human life of some of the worst atrocities of World War II.
The report concentrated on quackery’s effect on the elderly, and consequently focuses especially on cancer and arthritis. Yet the author makes clear that quackery defrauds all ages and extends to many other diseases and conditions. In an exhaustive search for information, the author’s investigators gathered information from federal agencies, such as the FDA, the FTC and the Department of Justice, as well as state and local sources, like state Attorneys General and law enforcement. They also contacted charitable organizations focused on specific diseases, like the American Cancer Society, consumer advocates and individual victims. Health claims were evaluated by medical and scientific experts and their conclusions included in the report.
“Plus ça change, plus c’est la même chose“
The author found many quack beliefs and practices that will be familiar to SBM readers.
    • The idea that many diseases are caused by vitamin and mineral deficiencies and a “more is better” approach to vitamin and mineral supplementation.
    • An unwarranted belief by the public that the health field is rigorously policed, which gives undue credibility to quack claims.
    • The appeal of quack remedies for diseases for which medicine sometimes offers limited options and no absolute cures, as well as conditions which result in chronic pain.
    • The claim that food allergies are the cause of diseases that have nothing to do with allergies.
    • The putative value of plants and herbs to prevent, mitigate or cure disease, even in cases where these remedies are unproven or disproven, and sometimes dangerous.
    • The idea that the body stores “toxins” which must be eliminated by enemas and herbal products including colon, gall bladder, liver and kidney “cleanses” or “flushes.” The colon is depicted as particularly vulnerable: “death begins in the colon” and the colon is incapable of proper operation without various interventions to “detoxify” the body.
    • Copper or magnetic bracelets as a cure or preventative for arthritis.
    • Treatment consisting of “adjusting the segments of the spine to correct vertebral distortions interfering with the nerve supply necessary for normal functioning of the organs and tissues of the body, i.e., correction of ‘vertebral subluxations.’”
    • A seemingly endless variety of weight-loss schemes.
    • Improper “energy flow” and other nebulous “imbalances” as the cause of disease.
    • Quack cancer clinics (in the U.S. and abroad) and treatment methods, including Hoxey treatment and Gerson therapy, as well as accounts of parents who opted for quack therapies for their children in lieu of conventional care.
    • Bogus diagnostic devices, including computerized nutrition analysis, boxes of colored lights, non-standard blood diagnostics and galvanometers.
    • The idea that “your disease is your fault” and that a better attitude, happy thoughts or a more positive outlook would improve the patient’s condition.
    • Surgery, radiation and chemotherapy as “unnatural” therapies to be avoided, and outright advice to cease prescribed medical treatments.
    • Fasting and complicated diets as a preventative and cure, such as macrobiotic and raw food diets, including copious servings of wheatgrass juice. One practitioner’s view was that “doctors are murderers and hospital food is poison.” (OK, maybe he was onto something with that last one.) Certain foods are to be avoided altogether, even if the patient is not allergic to them: coffee, those with food additives and flour, among others. Two popular myths that underlie these diets are that all diseases stem from a faulty diet and that sub-clinical deficiency diseases abound in our populations.
    • Exorbitant prices charged for worthless cures.
    • The use of testimonials and case studies as evidence of effectiveness.
    • Water as a cure all: “One of the most prominent themes in the mythology is that water is a cure for arthritis and/or other diseases. This applies to water which is consumed by mouth, taken by injection, sprayed with a nozzle on parts of the body or in which the joints are to be immersed. There is virtually no limit to the permutations. It can be hot water, lukewarm water, cold water, ice cold water or solid water the form of ice and can be salt water, mineral water, distilled water, spring water – the list is endless.”
    • Other treatments such as reflexology, bee pollen, coffee enemas and glandulars.
And the author is . . .
It wasn’t us, although we consistently make many of the same points.
This comprehensive report, entitled “Quackery: A $10 Billion Dollar Scandal” was issued by the Subcommittee on Health and Long-Term Care of the House of Representatives Committee on Aging in 1984. The effort was promoted by the Subcommittee’s Chairman, Claude Pepper (D-Florida), who made advocacy for the elderly the centerpiece of his political career. The report is significant not so much for its findings, as interesting and as seemingly contemporary as they are, but rather for its attitude toward pseudoscienctific health care products and practices. What thirty years ago was called “quackery” is known today as alternative, complementary and integrative medicine and promoted by the very government that was so deeply concerned about its deleterious effects on the American public in 1984.
And it is not only the government that has devolved from science to pseudoscience since the report came out. The medical experts who evaluated and roundly criticized quack practices came from the very institutions that have allowed these same practices to infiltrate their operations: Georgetown University Medical School, the Cleveland Clinic, the University of Maryland, Harvard, and the Mayo Clinic. Compare, for example, the Report’s conclusion on acupuncture with integrative medicine’s current enchantment with this “ancient” Chinese remedy.
There has been a great deal of skepticism in the United States among the traditional medical community as to the effectiveness of acupuncture. Western experts have traveled to China to study the ancient art. For a while, acupuncture seemed to be finding support within the medical community, but the results of carefully controlled tests have turned the experts away from it. For more than a dozen years the Arthritis Foundation did not take a position on acupuncture. Its position now is as follows: We now have enough data from solid scientific work to express a view about acupuncture. Three studies – one on chronic shoulder pain, one on osteoarthritis and one on rheumatoid arthritis – have demonstrated that the beneficial effects of acupuncture are: (1) only on the symptoms of arthritis and not on the disease process itself; (2) due entirely to the placebo effect; and (3) are short lasting.
The Report then goes on to describe these studies, which compared real acupuncture given to one group with a control group, for whom the needles were “placed only lightly on the skin and, instead of being inserted in the traditional points, were positioned all over the body.” The result? The amount of pain relief was the same. Thus, “there is no good evidence that acupuncture is of any specific value in treating any form of arthritis.” Of course, as the Report noted, there are nevertheless “promoters who have sought to seize on the oriental mysticism of acupuncture.” With these results, one wonders why acupuncture continued to be studied in trial after trial (including additional studies on osteoarthritis) for the next 30 years, all with the same result. Can you imagine that over 30 years of studies of a particular surgical procedure, all negative, would nevertheless result in continued promotion of the very same procedure?
Some devices covered in the report seem like caricatures of quack inventions. No longer available (to my knowledge) are the Dynatone Facial Exerciser (for wrinkles), Rado Pad and Uranium Wonder Gloves (radioactive gloves for the cure of arthritis), the Radon Generator (produces radon gas, to be mixed with water), Zerret Application (a dumbbell-shaped device filled with “zerret water,” to be held for 30 minutes while the “healing rays expand all the atoms of your being,”) or the Mark Eden bust developer (“a clam-shaped tension device that supposedly exercised the breast” to make it bigger). (Remember ubiquitous ads for these in women’s magazines of the day?) I would note, however, that none of these devices are any more implausible than reiki, therapeutic touch or homeopathy.
While the technology of pseudoscience may have changed since 1984, the toolkit of the quack is remarkably similar. These guidelines from the Arthritis Foundation are offered for spotting the unscrupulous promoter :
     1.  He may offer a “special” or “secret formula” or device. . .
     2.  He advertises . . . “case histories” and testimonials from satisfied “patients.”
     3.  He may claim to know the “cause” of arthritis and talk about “cleansing your body of “poisons” and “pepping up” your health. [Today the language would likely be “eliminating toxins,” “increasing your energy” or “boosting your immune system.”] He may say surgery, x-rays and drugs prescribed by a physician are unnecessary.
     4.  He may accuse the “medical establishment” of deliberately thwarting progress, or of persecuting him . . . but he doesn’t allow his method to be tested in tried and proven ways.
The report displays none of the reverence for alternative practitioners that you find in government or in integrative medicine practices today. In fact, it puts them on a continuum that includes witches:
The tradition of healing which combines elements of religion, magic and medicine has probably always existed in various parts of the world. In modern times, these techniques range from the practices of medical mountebanks and “alternative” health professionals to witches, faith healers, nutritional faddists and spiritual healers.
Unfortunately, in the current “modern times,” “these techniques” are found in academic medical centers which form “integrative practices” with the very same alternative health professionals and nutritional faddists, except that we now call the latter naturopaths. Speaking of nutritional faddists/naturopaths, who claim they invented prevention through healthy lifestyle practices, look at these thirty-year-old (at least) dietary guidelines from the American Cancer Society, reprinted in the report:
     1.  Avoid obesity.
     2.  Cut down on total fat intake.
     3.  Eat more high-fiber foods, such as fruits, vegetables and whole-grain cereals.
     4.  Include foods rich in vitamins A and C in the daily diet.
     5.  Include cruciferous vegetables such as cabbage, broccoli, Brussel sprouts, kohlrabi and cauliflower.
     6.  Be moderate in consumption of alcoholic beverages.
     7.  Be moderate in consumption of salt-cured, smoked and nitrite-cured foods.
The Subcommittee made several suggestions to alleviate victimization of the public by quacks: increased government enforcement efforts, increased criminal penalties, more funding for research into diseases (such as cancer) which make patients particularly vulnerable to offers of miracle cures and other pseudoscience, more consumer education, strengthening medical licensing statutes to make practice of medicine without a license a felony and more patient protection when non-standard therapies are offered.
The only suggestion for reform apparently taken up by future Congresses was to establish a federally-sponsored mechanism to evaluate evolving medical technologies and procedures for safety, efficacy and cost-effectiveness. Unfortunately, the purpose of the “federal mechanism” actually created by Congress, the National Center for Complementary and Alternative Medicine, was to “validate” alternative medicine. All in all, you’d almost think the state and federal governments spent the last 30 years in a deliberate attempt to thwart Rep. Pepper’s and the Subcommittee’s goals. They certainly did a good job of it.
In its conclusions, the Report takes on a criticism often directed at those who support science-based medicine: that we automatically reject anything that doesn’t originate with conventional medicine. It responds in almost the same terms we have used.
Does that mean that the Committee condemns each unproven remedy mentioned in the report?
No. Many of the so-called “cures” listed in this report have been proven false. Others are so ludicrous that the commonsense indicates they cannot be of much value. Others have a scintilla of credibility. It may be that the next major breakthrough in medical science is listed in this report, although that is highly doubtful. What the Committee is saying [is] that a cure or remedy, before it can be proclaimed as such, must be proven to be so through the established procedures of the scientific method. There must be unbiased, scientific evidence that a remedy is helpful and that it does not provide harm or untoward side effects before society can allow its widespread use by the general public.
We have gone terribly astray in the last 30 years. What went wrong?

2015 NHIS Report on Complementary Health Approaches (whatever that means)
Jann Bellamy on February 19, 2015
Link


Back in 2004, data from the 2002 National Health Interview Survey (NHIS) appeared in a report titled “Complementary and Alternative Medicine Use Among Adults: United States, 2002.” It showed a whopping 62% of adults had used CAM in the past 12 months, but only if prayer for health reasons was included. With prayer excluded, the percentage was substantially lower, at 35%.
“CAM” was defined as:
a group of diverse medical and health care systems, therapies, and products that are not presently considered to be part of conventional medicine.
The authors noted that, in earlier surveys of CAM use, “CAM has been operationally defined in a variety of ways” and the lists of CAM interventions/therapies included “varied considerably among the surveys.”
The most commonly used CAM therapies (excluding prayer) were non-vitamin, non-mineral natural products (18.9%), deep breathing exercises (11.6%), chiropractic care (7.5%), yoga (5.1%), massage (5.0%) and diet-based therapies (3.5%). CAM was most often used to treat back pain or problems, head or chest colds, neck pain or problems, joint pain or stiffness, and anxiety or depression. Most CAM use was self-prescribed. Rebranding things like exercise (yoga) as “CAM” was in the mix from the get-go.
Non-vitamin, non-mineral natural products used most frequently were Echinacea, ginseng, Ginkgo biloba, and garlic supplements. (I am leaving out percentages because the period covered, as well the way of calculating percentages, changed from year to year, and I gave up.)
Use of acupuncture (1.1%), homeopathic treatment (1.7%) naturopathy (0.2%), and energy healing (0.5%) was miniscule.
All in all, the authors found the results “surprising given the lack of definitive evidence supporting the safety and efficacy of most CAM interventions.”
In 2008, analysis of more recent data from the NHIS was published: “Complementary and Alternative Medicine Use Among Adults and Children: United States, 2007.” This time, CAM was defined as:
a heterogeneous spectrum of ancient to new-age approaches that purport to prevent or treat disease. By definition, CAM practices are not part of conventional medicine because there is insufficient proof that they are safe and effective.
Or, in more condensed language, CAM was defined as practices that have insufficient proof of safety and effectiveness.
Several changes were made in both the definition and reference period for some CAM treatments. More “natural” products were included and their reference period was reduced from 12 months to 30 days. “Chiropractic care” became “chiropractic or osteopathic manipulation.” Children were included for the first time.
Overall use by adults jumped to 38%. Yet, the therapies most commonly used by adults and their percentages didn’t change all that much: non-vitamin, non-mineral, natural products (17.7%), deep breathing exercises (12.7%), meditation (9.4%), chiropractic or osteopathic manipulation (8.6%), massage (8.3%), and yoga (6.1%) were at the top of the list.
Adults most often used CAM for the pretty much the same things as the 2004 article reported: back pain or problems, neck pain or problems, joint pain or stiffness or other joint conditions, arthritis, anxiety and depression, and other musculoskeletal conditions. However, use for head and chest colds decreased markedly.
Of the top 4 natural products, only Echinacea remained. Fish oil, omega-3 fatty acids and DHA were at the top of the list. Glucosamine and flaxseed oil or pills made the top four as well.
Homeopathic treatment (1.8%), naturopathy (0.3%), energy healing (0.5%) and acupuncture (1.4%) still showed very little use among adults.
The most interesting insight of the 2004 Report was not so much CAM use as it was CAM use versus CAM research:
The attention given by the scientific community does not correlate with the prevalence of use by the public as measured by the 2007 NHIS. For example, the Institute of Medicine identified 79 systematic reviews of acupuncture and 38 studies of homeopathy, placing them 3rd and 4th among all CAM therapies. Yet NHIS found that less than 1.5% of the adult U.S. public used each of these therapies in a given year. In fact, there is no meaningful correlation between the number of published studies of a CAM therapy and its use by the U.S. public. CAM therapies with relatively infrequent use by the public (e.g., biofeedback, hypnotherapy, acupuncture) are those with the highest level of acceptance and referral by physician groups, including pediatricians.
On February 10, 2015, a new Report was unveiled: “Trends in the Use of Complementary Health Approaches Among Adults: United States, 2002-2012.” A separate Report covered children.
Once again, complementary medicine was redefined. In fact, it’s not even “medicine” anymore. With no explanation for the change that I could find, complementary and alternative medicine is no more; we are now looking at “complementary health approaches,” which are
an array of modalities and products with a history of use or origins outside of conventional Western medicine.
The Report includes “revised prevalence estimates for 2002 and 2007 [which] were lower than those published previously: 32.8% compared to 35% for 2002 and 35.5% compared with 38% for 2007.” This is explained by a “narrower definition,” but of what and how it is “narrower” I am at a loss to find. In any event, the figure for 2012 is 34%.
The authors do note that a “confusion with definitions” about “which complementary health approaches are included in the definition of any complementary health approach” [got that?] has led to “an effort to establish an internationally accepted standard for what approaches should be included in prevalence surveys of complementary health approaches.” I take it this goal remains unrealized.
“Alternative” and “medicine” were not the only things dropped. For reasons that are not explained, statistics regarding, or discussion of, the diseases or conditions for which consumers are using these “health approaches” is missing from the article. That is not to say it might not be somewhere in the information gathered in the survey, but it is a mystery why the authors wouldn’t mention it. After all, as the authors state, these surveys were designed to help guide the NCCIH (f/k/a NCCAM) research agenda, which, according to the authors, has now shifted from disease treatment to symptom management and “the promotion of optimal health,” whatever “optimal health” might be.
So what health approaches are American adults using most often? Surprise! It’s non-vitamin, non-mineral dietary supplements (17.7%), deep-breathing exercises (10.9%), yoga (9.5%), chiropractic or osteopathic manipulation (8.4%), meditation (8.0%), and massage therapy (8.0%). After that, special diets come in at 3.0%, with all others at 2.2%, for homeopathic treatment, or below. The “below” includes acupuncture (1.5%), energy healing (0.5%) and naturopathy (0.4%).
The top four supplements were fish oil, glucosamine or chondroitin, probiotics or prebiotics, and melatonin.
I find these Reports both interesting and disturbing. Obviously, it is interesting to read what healthcare practices people are using. But the disturbing elements predominate.
The definitional calisthenics on display over a decade of NHIS surveys of alternative/complementary medicine/healthcare approaches is simply a symptom of the real problem underlying the whole enterprise, one that has vexed us here at SBM since the beginning. Kimball Atwood tackled it early on in two posts documenting the slide from “quackery” to “CAM.” More recently, David Gorski blogged about yet another definitional reboot when the National Center for Complementary and Alternative Medicine changed its name to the National Center for Research on Complementary and Integrative Health, not long after Mark Crislip weighed in on the “loose” terms of SCAM.
I’ll add my observations, if not a better understanding or a solution.
The root cause, if I may use that term, lies in the uselessness of a “Western conventional medicine”/”not Western conventional medicine” dichotomy. The real distinction to be made, and the only one that is useful, is one of science and evidence.
“Western conventional medicine” is characterized by the rational application of science to healthcare practices, one feature of which is determining efficacy and safety, as best we can, in advance of using any particular treatment. But “Western conventional medicine” doesn’t have to be the exclusive repository of those characteristics. Any healthcare discipline or practice can take up science and evidence as its standard. Acupuncture, chiropractic and naturopathy are, in varying degrees, Western, conventional and medicine, but none is all of those. Yet any of these disciplines is more than welcome to employ science and evidence, as are the dietary supplement or homeopathic industries. But they don’t, or at least not consistently or reliably.
That is the key difference and that is the distinction that must be made when discussing these practices, instead of using some fabricated terms that paper over what is really going on. NCCIH and the NHIS Reports admit that evidence is an issue, but NCCIH doesn’t want to make it clear that it is also the difference that really matters. And by the way, it’s unfortunate to see the NHIS dragged into this by NCCIH’s apparently false assurance that the results would “guide” their research agenda. If that is truly the case, why so much acupuncture research? According to the NCCIH website, they are still into acupuncture, now in cahoots with the military:
the report proposes expanded collaboration between NCCIH and the U.S. Department of Defense (DoD) and the Department of Veterans Affairs (VA) to test approaches to pain management that integrate complementary treatments such as acupuncture, massage, and meditation with conventional models of care.
Because of my interest in CAM practitioner licensing and other legislation affecting CAM practices, such as DSHEA, I was also struck by the tremendous gap between what these laws permit and what people are actually doing. Why are legislatures defining the chiropractic scope of practice to include the diagnosis and treatment of almost any disease or condition when, by far, the only thing people use them for is a limited range of musculoskeletal problems? Why do so many states license acupuncturists and give them leeway to use all sorts of nonsense on patients when hardly anyone uses them for anything? And why do states continue to fall for pleas for naturopathic licensing when there is no demand? Why the hassle of a regulatory board and the other accouterments of bureaucracy to accommodate a handful of people?
Finally, consumer demand as an excuse for “integrative medicine” is a bunch of hooey. The idea that patients are “demanding” acupuncture is just nonsense. Looks like some physicians are the ones demanding acupuncture and acupuncture research, in spite of, not because of, patient demand. Other popular integrative medicine center practices, such as cranial sacral therapy, reiki and energy healing are registering at below 3%.
I’ve come to believe that CAM and “integrative medicine” is mostly about money and ideology, not patients. Money from research, money from academic and government jobs beholden to perpetuating CAM, money from selling CAM products and services, money from education and training in CAM. Fortunately, despite the efforts of the many people whose livelihood depends on CAM, the public seems remarkably resistant to their efforts. They tend to cluster around treatments that, if not supported by sufficient evidence of safety and effectiveness, at least have some plausible potential for helping them, even it turns out to be an unrealized potential: spinal manipulation for musculoskeletal issues, supplements like fish oil, relaxation techniques like deep breathing and exercise practices like yoga. It is amazing that so few have been fooled by so many who are trying so hard. It’s not ideal, it’s not an excuse to tolerate CAM, and it’s not a reason to become complacent, but it does look like consumers are smarter than the ideologues and money makers give them credit for.

Media Literacy Is Key
Media literacy is an important component to teaching science and critical thinking. We’ll add that to our to-do list.
Steven Novella on June 26, 2019
Link

Educating the public about medical myths and misconceptions has various challenges. The psychological deck seems to be stacked against us. It’s easier to scare people with possible risks than to reassure them with facts. People tend to be more compelled by emotional anecdotes than dry data. There is something inherently compelling about conspiracy theories that attract many people. People are good at remembering dramatic details, but poor at remembering whether or not they are true and what the source of the information is.
But perhaps the most profound factor making our job difficult is that once an idea has taken root in someone’s mind, it is remarkably difficult to change. Humans are instinctively good at motivated reasoning and confirmation bias. We see what we want to see, remember the bits that support our narrative, and can rationalize away pesky things like logic and evidence. The result can be a powerful, even overwhelming, illusion of confident knowledge, even in notions that are patently absurd. We can then erect elaborate defenses around these beliefs to protect them from reality.
This is all bad enough, for ideas that spread naturally through the culture, but these cognitive bugs can also be hijacked by those with an agenda, which is usually to sell something. This is how pre-packed narratives create and maintain a market for useless and implausible snake oil and dubious medical treatments. We can point out that homeopathic potions have no active ingredients and essentially break the laws of physics, and the response by defenders is the equivalent of, “Ignore that man behind the curtain”. Just say something about Big Pharma, shills, and then coat it with some hand-waving nonsense about water memory, and belief in magic is maintained.
At this point you’re probably thinking – this is all just the setup, right. What’s the solution? There is a solution, actually. Not a silver bullet, but at least something that works. It’s clearly difficult (I won’t say pointless) to give people information after they have already been indoctrinated into pseudoscience and conspiracy theories. So obviously we need to get to them before misinformation has taken root. This is where education comes in, and not just of children in school, but lifelong learning.
There are basically three types of preventive education that are likely to reduce susceptibility to pseudoscience. The first is scientific literacy. There is some controversy over how effective this is, however. A few decades ago the “knowledge deficit model” was dominant, and the prescription for belief in pseudoscience was to teach people science. However, recent research has not been kind to the knowledge deficit model. You cannot usually change someone’s mind about an emotionally held belief with just information.
But sometimes you can. You can move the needle a bit with public education about certain topics. Sometimes beliefs are based more on misinformation than emotion or identity, and if you correct that misinformation you can change beliefs. This is very topic specific, and also is affected by the kind of information you give and how it is presented. For example, global warming denial is strongly predicted by political ideology, and not at all by scientific literacy. However, vaccine denial does correlate with low scientific literacy, which implies that science education can be a mitigating factor.
Other research, however, shows that simply giving facts is not enough. It is more effective to replace one narrative with another explanatory narrative. This is because our knowledge is not just a collection of facts. We weave those facts into an explanatory narrative that we use to understand the world, which gives us at least the illusion of control and predictability. If you tell someone a fact that challenges their explanatory narrative, you are attempting to pull the carpet out from under them. They will resist (with fervor proportionate to the degree to which they are emotionally invested in the narrative). But – if you simultaneously give them an alternate narrative, they can swap one out for the other without losing their equilibrium.
I am not suggesting this is easy, or always works, but it appears to be statistically more effective than just doling out facts.
The second type of education is critical thinking. This relates more directly to the point about narratives. Critical thinking is about metacognition, knowing how to think with a valid process that is self-reflective and therefore potentially self-corrective. If someone understands exactly how conspiracy thinking is a cognitive trap, they are less likely to fall into that trap. Critical thinking and scientific literacy is a powerful combination – this is essentially what we mean by scientific skepticism, which is exactly what we are doing here (in the realm of medicine).
But there is a third component to education that arguably sits on top of the other two – media literacy. This is something scientific skeptics have addressed tangentially, but increasingly is being recognized as a primary concern we need to take head on. It’s easy to conclude that this is due to the rise of social media, but of course this issue has always existed. It is, however, exacerbated by the fast and open exchange of information and ideas allowed by the World Wide Web.
While we should focus attention on formal education, most people get most of their knowledge and information outside the classroom. Consumption of information is increasingly online, although traditional media still dominate. We live in a multi-media world, and those with access are constantly consuming information from multiple sources, increasingly online. Trying to change what people believe with formal education is therefore a losing proposition. We need to engage everywhere.
But this also implies that media literacy is critical to overall information literacy. I won’t say it is more important than scientific and critical thinking literacy, because the three work together and cannot be disentangled. Media literacy may be the linchpin, however. Recognized principles of media literacy (again – with much overlap with critical thinking) include:
    • Question everything. Don’t believe something just because it feels right – especially if it feels right.
    • Think about the source of information. Is it a primary or secondary source? What are the likely biases of that source? Is it tied to an agenda? Is it authoritative?
    • Check multiple sources. Are all or most sources saying the same thing?
    • Try to track back to the original primary source, rather than trusting someone else’s summary.
    • Are the claims being made credible or plausible?
    • What does the actual evidence say? Distinguish this from how it is being interpreted, or what speculation is flowing from the data.
    • Are there any apparent attempts to manipulate your emotions? Are they appealing to any form of tribalism, fear, greed, or otherwise pushing emotional buttons? Are they using “click bait” headlines, or sensationalizing the facts?
All that is a lot of work, however. If we want the greatest number of people to actually do something we need to make it as easy as possible. This is again a significant inherent disadvantage to the skeptical position – it’s difficult, and hard work by its very nature. Meanwhile, going with your feelings and intuitions is much easier. There is, however, a way to make media literacy more intuitive, and that is through experience. This is the idea behind a recent study that uses a game to teach media literacy through experience.
The idea is to have subjects play a game on their smartphone in which their task is to manipulate social media in order to maintain credibility while fooling as many people as possible. This may seem counterintuitive – aren’t we teaching these subjects to be deceptive? But I think it’s like teaching someone stage magic. Magicians are famously skeptical, because they know how easy it is to deceive others. It’s their job. Knowledge of deception inoculates them against those same methods of deception.
That is what the researchers found with social media as well. Teaching people the mechanisms by which social media can be used to manipulate emotions, because they themselves are doing it, inoculates them against those same methods. Essentially the game teaches them to be skeptical critical thinkers when it comes to consuming information online.
The job of promoting science-based medicine is increasingly difficult, but at the same time our knowledge of how to do it optimally is at least improving. We now have a more clear idea that we need to focus simultaneously on scientific literacy, critical thinking skills, and media literacy. We also cannot just take comforting beliefs away from people. We need to show them how to make sense of our complex world with facts and logic, and how to avoid all the common cognitive pitfalls. And we need to promote media literacy, so that people can learn for themselves how to evaluate sources and information.
Sure, this is a herculean task. But the alternative is to slide numbly into a post-truth world of fake news, truth decay, pseudoscience, and alternative facts. There is no final or complete success of failure in this struggle. We’ll just keep pushing as hard as we can to move that rock up the hill.

The Scientific Attitude, Not the Scientific Method, Is the Key
A philosopher of science argues that science is not characterized by a specific scientific method but by the scientific attitude. Scientists value empirical evidence and follow the evidence wherever it leads. They are open to changing their mind rather than stubbornly clinging to an ideological belief system.
Harriet Hall on May 7, 2019
Link

Science is under attack. The evidence for global warming is overwhelming but many reject the evidence in favor of ideology, and just believe what they want to believe. Vaccine-preventable diseases are rebounding due to rejection of vaccine science and the circulation of egregious misinformation. There are many misunderstandings about science, even among scientists. In this “post-truth” world, we must be able not only to explain why claims based on scientific evidence have a superior claim to believability but also to persuade the public to accept them over competing claims that are not based on empirical evidence but are based only on ideology or wishful thinking. Lee McIntyre, a philosopher of science, has spent his entire career grappling with this problem, and in his new book The Scientific Attitude: Defending Science from Denial, Fraud, and Pseudoscience he tries to explain what science is by examining what it is not. He says “there is much to learn from those who have forsaken it.” He argues that “the scientific attitude” is the key, even though it is difficult to pin down.
There’s no such thing as “the scientific method”. The simplistic “observe, hypothesize, predict, test, analyze, and revise” model does not describe how most scientific discoveries are actually made. Penicillin was not discovered by systematically applying a “scientific method”, but by serendipitously profiting from an accidental contamination. The demarcation problem (drawing a line between science and non-science) has never been satisfactorily solved. Criteria like falsifiability showed promise but were flawed. McIntyre’s goal is to provide a way to show what is distinctive about science without solving the demarcation problem.
No matter how good the evidence, science can never prove the truth of any empirical theory. Theories are always tentative and subject to revision based on better evidence. But this doesn’t mean we have no grounds for believing a scientific theory. Evidence counts. Predictability is compelling. Good theories don’t require knowing the mechanism; we knew penicillin worked long before we understood how it worked. He draws a distinction between truth and warrant. A theory has warrant if it has a credible claim to believability given the evidence, even though we can’t prove it’s true in any absolute sense. We can be comfortable accepting a warranted theory even though we realize it might eventually be proven wrong. We can celebrate uncertainty, and we can require denialists and conspiracy theorists to defend the warrant of their beliefs rather than simply attacking our science.
Einstein had the true scientific attitude. He said experimental confirmation would not establish the truth of his theory, but he would have to accept that his theory was untenable if it failed certain tests. Specific methodology is less important than the values of honesty and openness. Individuals must be willing to critique their own ideas, but criticism is also a communal and institutional self-correcting enterprise. We try to find failure. Science can never be purely objective; it is value-laden. “It is a myth that we choose our beliefs and theories based only on the evidence.”
The scientific attitude transformed modern medicine
Semmelweis had the scientific attitude far ahead of his time; his ideas were met with unreasoned opposition. “Until quite recently in the history of medicine, patients often had just as much to fear from their doctor as they did from whatever disease they were trying to cure.” Benjamin Rush was a strong advocate of bloodletting to balance the humors; perhaps it was fitting that he died of bloodletting when it was used to treat his typhus fever. In his day, patients treated by a doctor for compound fracture had only a 50% chance of survival. Gradually doctors accepted that they could learn from experiments, the scientific attitude prevailed, and modern medicine was born.
According to the myth, penicillin was discovered by accident. It was no accident. Fleming responded to a chance contamination with a prepared mind, understood that it showed antibiotic properties, published his findings, and others tested and developed the drug. The individuals involved all had the scientific attitude.
Science gone wrong: errors and fraud
Science can go wrong in many ways: p-hacking, cherry-picking, sloppiness, laziness, cognitive bias, unintentional errors, intentional falsification or fabrication of data (sometimes with “good intentions” because they are sure they are right and want to speed the process along), and deliberate fraud, as in the case of Andrew Wakefield, whose now-retracted study on autism led to vaccine refusals and measles resurgence.
Denialists and pseudoscience
The most valuable chapter of the book is “Science Gone Sideways: Denialists, Pseudoscientists, and Other Charlatans.” Those people either misunderstand or don’t care about the standards of scientific evidence, or if they do care, they don’t care enough to modify or abandon their ideological beliefs. He defines denialism as “the refusal to believe in well-warranted scientific theories even when the evidence is overwhelming” and pseudoscience as when someone seeks the mantle of science to promote a fringe theory but refuses to change their beliefs in the face of refutatory evidence or methodological criticism. This is dangerous because people who have an economic, religious, or political interest in contradicting certain scientific findings “have resorted to a public relations campaign that has made great strides in undermining the public’s understanding of and respect for science.”
Denialists and pseudoscientists may believe that they are living up to the highest standards of the scientific attitude, but they are not. When the evidence conflicts with a sacred belief, people who believe they already know the answer will reject science. Superstition and willful ignorance are not new, but what is new is “the extent to which people can find a ready supply of ‘evidence’ to support their conspiracy-based, pseudoscientific, denialist, or other outright irrational beliefs in a community of like-minded people on the Internet.” It is easy to avoid conflicting views and to live in a fact-free alternative reality. Group consensus works well in science as a check against error, but outside of science charlatans use consensus as a way to reinforce prejudice.
Denialists like to call themselves skeptics, and claim to be open to new ideas, but true skeptics reject new ideas that are believed without sufficient evidence and change their minds to adopt new ideas that are based on good evidence. “For denialists, no evidence ever seems sufficient for them to change their minds.” This is not a scientific attitude; it is the opposite of having an open mind. They demand impossibly high standards for disconfirming evidence but demonstrate complete gullibility for any claim that supports their belief system; for instance, they don’t need evidence to believe the claim that the CDC paid the Institute of Medicine to suppress the data on thimerosal.
Denialists and pseudoscientists like to point out that cranks can sometimes get it right. They laughed at Galileo, but Galileo was not a crank; he had evidence to support his assertions. The book tells the story of J. Harlen Bretz, a maverick geologist who found evidence that a massive flood had created the channeled scablands of Eastern Washington. He realized that what he saw could not be explained by uniformitarianism, the generally accepted dogma that geologic processes acted gradually over time. He didn’t know what could have caused such a mega-flood, but others found the explanation: the failure of a giant ice dam at prehistoric Lake Missoula. He was long considered a heretic, but his evidence was overwhelming. Others could see the evidence for themselves. Bretz was vindicated in 1965 when geologists went on an official field trip and sent him a telegram saying, “We are all catastrophists now.” It took a while, but the scientific attitude eventually triumphed.
“The crucial question for pseudoscientists is this: if your theories are true, where is the evidence?” If you have good answers, you will not be persecuted or ignored; science will eventually beat a path to your door.
Conclusion: a worthy project
McIntyre says the core difference between what is science and what is not hinges on the scientific attitude. He hopes that his concept of the scientific attitude will put us in a better place to understand and defend science and to grow it in new fields like the social sciences. Truth matters, and if we care about empirical evidence and use it to shape our theories, we can avoid remaining “mired in the ditch of ideology, superstition, and confusion”. Our very survival as a species may depend on it.

Guiding Lights
Mark Crislip on September 7, 2012
Link 
This is, I admit, a content free post. July and August are the sunny days here in the great Pacific Northwest, and rather than spend time in front of the computer, I am outside with the kids. To compound matters, I was on call the labor day weekend (I usually write the first draft the weekend before the posts are due) and was very busy. I am finishing this early on Thursday on an airplane to Vegas. My wife and I are taking our first non-child containing vacation in 19 years while my youngest is on a 4 day school trip. Wander the strip, see a show and enjoy the desert heat as a couple and not a family.
I have not had the time to spend researching a topic, so instead I thought I would ramble on about 2.5 topics that have been on my mind. Writing helps to focus my thoughts. Even though I often have residents on service, I still write daily notes as the act of putting thoughts into words is the best way to clarity thoughts. Next week the kids are back at school and I am sure the rains will start up and I will again have time to go into full research mode. In the meantime feel free to ignore this post.
There is nothing to see here. Move along.
As a third year resident I came across the notes I had written on a patient when I was an intern and I thought, man, What a maroon, what an ignoranimus. I was amazed what I had learned during the prior three years. Somewhat of the same process has occurred in my time at SBM. Not only have I learned a ton about various SCAMs, but more importantly I have refined and extended how and why I think.
There were two events recently that started my unusual attempt at introspection. Socrates I am not, preferring to keep the demons at bay by keeping my mind busy with external stimuli rather than examine my own life. The first was the entry on the central dogma of SCAM by David Gorski. The second was a video from TAM, where Jamie Ian Swiss talked in part about the criteria for being a skeptic*. It lead me to thinking about the criteria I use to assess medicine and the medical literature. What are the standards I mostly try to follow? They are not dogma so much as the first or guiding principles I use. I write here only for myself, my colleagues are free to disallow any knowledge of my actions.
There is a reality independent of human existence
It is a truth I hold to be self evident. If all humans were to vanish in a puff of CO2 and H2O, reality would go on without us. Long after the sun has gone nova Voyager will continue its trajectory through space following the laws on motion and gravity. Unless it returns as V’Ger.
Reality is defined by the basic sciences
At the human scale, classical physics, anatomy, chemistry, physiology, biochemistry etc. provide a reasonably complete framework for understanding medicine and reality. If there is, as someone more fluent in English than I suggested, more things under heaven and earth than are drempt of in my philosophy, then it needs to be demonstrated as either an extension of what we know or as radical new physical phenomena. In my scant 30 years in medicine all the breakthroughs have come from extensions of what we know. The demonstration of energies, physiologies and anatomies that are the underlying mechanism by which the various SCAMs are purported to have their effect are a Godot yet to arrive.
Plausibility within the context of known reality is important in evaluating the truthiness of new medical phenomena
I have practiced medicine for almost 30 years and while there has been much that has been learned or discovered, there is nothing that has not been within the framework of the basic sciences. In contrast, much of SCAM is not just implausible but impossible given the accumulated knowledge of the last 500 years. There is unlikely to be medical discoveries the equivalent to the dark matter and dark energy. The dreams of Oz et al for a new medicine are illusory.
The default mode of the brain is not rational/critical thinking
Not only is it difficult to think rationally and avoid the numerous logical fallacies to which we are prone, I wonder if it is impossible for some.
Most human characteristics fall into a bell shaped distribution, with some having more or less than others. Variability is the norm, although within often narrow parameters. If there is a spiritual module in the brain, I do not have it. Whether I was raised that way or born that way, I cannot say, but it seems to be X-linked in my family. I have always been unable to comprehend spiritual concepts, and it was not for a lack of trying in younger days. All I ever developed from meditating in a Zen temple were numb legs from sitting cross legged. I have friends and family who seem to commune with a god or spirits I can neither understand or comprehend. It is, I suppose, like color to a deaf person or music to the blind^.
Similarly, I really wonder if a rational/critical thinking module is missing or rudimentary for some people. How else can someone read about homeopathy or reiki and not laugh? Or maybe it is me. I lack the necessary neurologic module that can see the truth.
The first principle is that you must not fool yourself, and you are the easiest person to fool. ~ Richard Feynman
Combined with the Dunning-Kruger effect, this makes for a potent combination for researchers, patients and physicians to completely misinterpret information. My time in the world of SCAM has reinforced the notion that we think poorly, do not realize that we think poorly and are unable to realize that we can’t realize that we think poorly. Convoluted? I will not go another level into the Inception. N-Rays remains my archetype example of how easily it is for honest and intelligent researchers to fool themselves and others. So much of the SCAM literature is measurement N-Rays emitted from the tooth fairy. I think Harriet is unaware that tooth fairies are the primary source of N-Rays.
We are prone to a plethora of logical fallacies that keep us convinced that unreality is real and those same fallacies keep us from recognizing our errors. Perhaps that is why I do not like to examine my own life; the enormity of the mistakes could cause me to explode like Mr. Creosote.
Most medical research is of poor quality and the results either overstated or wrong
Although there is a great deal of enjoyment in dissecting a SCAM paper, most being of remarkably poor quality, I have come to realize how much of what is published in medical literature is suboptimal. Quality medical research is extremely difficult to do and that is certainly reflected in the SCAM literature, where it is rare to find an article that doesn’t have all the criteria that make the conclusions suspect.
It is sobering that simultaneously we know so much and so little about the treatment of common diseases. The nice thing about infectious diseases is I have the luxury of knowing that if I can kill a bug in the test tube, I can usually kill it in the patient. It makes treatment decisions easier when there are a lack of well controlled clinical trials for unusual diseases.
The plural of anecdote is anecdotes, not data
Anecdotes are powerful. People respond to stories of cures more viscerally than knowing the results of a study. And there has never been a good answer when a believer tells me an anecdote where their friends aunt had pain relief from acupuncture or a cancer cured by colloidal silver, just what do you have to say to that, Mr. Skeptic? Nothing, of course, that doesn’t make me look like an out of touch, bloviating, old naysayer, droning on about bias and placebo.
We do need to be better with anecdotes and stories. Not as proof of a concept, but as illustrative of a concept. In medicine lectures are often started with a case to personalize the disease or treatment to be discussed.
In. My. Experience.
They remain the three most dangerous words in medicine, at least as a criteria for choosing a treatment. A physician who uses the words ‘strong’, ‘big gun’ or ‘powerful’ to describe antibiotics has demonstrated with almost 100% sensitivity and specificity they know nothing about antibiotics and should be ignored. Similarly, avoid the physician who chooses therapies from experience. Simultaneously, experience has made me an infinitely better diagnostician, but I would wager that most health care workers conflate the two .
Like anecdotes, however, ‘in my experience’ is the proverbial two edged sword. Patients want to know that you have experience and success with whatever treatment you are suggesting, and are less interested in what the studies show. I often try and mention both options: the studies demonstrate that I am prescribing the best therapy and it has worked in my hands. Most of the time they are more reassured that I have experience than the fact I am up to date on the literature. Every now and then I will treat a disease for the first time, such as a tropical parasite. No one in Portland has extensive experience with, say Leshmania. Patients never look comfortable when I say this is my first case, but the data suggests the treatment will work. Physician and patient never think about the nuances of both anecdote or experience.
Understanding is always tentative and subject to change
Everything is subject to change, although the requirements for changing an opinion are proportional to the amount of prior information to support a conclusion. It would take an compelling study to have me change my mind that beta-lactams are the best antibiotic for the treatment of S. aureus endocarditis, or that evolution is false or that acupuncture effect is anything besides patient and physician reporting bias.
We are now on our descent into the Las Vegas airport. That’s all folks. .Those were all of my opinions. Thank you for reading them.
*Digression.
I came across an interesting quote by way of Skepchick in the Humanist. As a side note in an article on women in the secular movement, Susan Jacoby make an interesting comment:
In the Center for Inquiry, the organization with which I’m most familiar, this often expresses itself as a division between “humanists” and people who call themselves “skeptics.” There is a lot of overlap between these two groups but, in my experience, the skeptics tend to be more conservative…
My first thought was Hm. Is that the case? That is not the distinction I would have made. Conservatives, to paint with a broad brush, are reality and data adverse. I tend to self label as a skeptic, and although I agree with most humanist positions, I never think of myself as a humanist.
What I know of both groups is more from their publications than from hands on experience. I usually find the Skeptic Inquirer interesting and informative and I usually find Free Inquiry to be uninteresting. The difference to my mind is that the former is more about facts, the way the reality may or may not function. The latter is more about opinion, the arbitrary laws, rules, habits and regulations that make a society more or less tolerable. What interests me is knowing how the natural world works. While I have strong opinions on the issues raised by Free Inquiry, the opinions are based at their root on nothing, and as such are not that interesting.
When we were playing golf last week my son double hit the ball. He wondered why it counted as two strokes and a swing and a miss also counted as a stroke. What, he asked, is a golf stroke? The swing or the contact with the ball? I told him both, it is one of the many arbitrary and often inconsistent and irrational rules that govern life. A humanist would worry about those rules, a skeptic about how to measure a double hit or a miss. While I agree with the humanist philosophy, I suppose I am more of a Shruggie when it comes to their application.
It would be an interesting sociology/psychology study to compare and contrast the two groups.
^ not a mistake.

Hostility towards scientific consensus: A red flag identifying a crank or quack
Deniers of well-established science, such as the anthropogenic climate change, the theory of evolution, and vaccine science, frequently attack the scientific consensus. Some go farther and attack the very concept of a scientific consensus. When you see such attacks, you know you’re dealing with a crank or, in the case of medicine, a quack.
David Gorski on December 15, 2014
Link

I have yet another grant deadline to deal with, this time for the Department of Defense Congressionally Directed Medical Research Programs, this time around its Breast Cancer Research Program. Unfortunately, that put a high degree of time pressure on me. Fortunately, there’s still stuff in the archives of my not-so-secret other blog that I deem quite appropriate for this blog and that can be updated with minimal effort. If you don’t know what I’m talking about when I refer to my not-so-secret other blog, then it’ll definitely be new to you. If you haven’t been reading that blog for at least four and a half years, it’ll be new to you as well. And even if you have seen it before, I think it’s worth revisiting.
Why? It came up because of an encounter I had on Twitter with Jane Orient, MD, who, as you might recall, is the executive director of the American Association of Physicians and Surgeons (AAPS). I’ve written about the AAPS before. You can get the details in the link, but if you don’t have time suffice to say that it is an entire organization of libertarian-leaning “brave maverick doctors” who think Medicare is unconstitutional, don’t believe that the government should have much, if anything, to do with regulating the practice of medicine, and reject evidence-based guidelines as an unholy affront to the independence of the physician. Along the way, the AAPS, through its journal, The Journal of American Association of Physicians and Surgeons (often abbreviated JPANDS), promoted antivaccine views, including the discredited concept that vaccines cause sudden infant death syndrome, HIV/AIDS denialism, and the scientifically unsupported idea that abortion causes breast cancer (a topic I might have to revisit, given the activity promoting it recently).
In any case, two or three weeks ago, I was having a bit of an exchange with Dr. Orient over anthropogenic global climate change (often abbreviated as AGW, for anthropogenic global warming, for short), the well-accepted science that concludes that CO2 generated by human activity is having a serious warming effect on the earth’s climate. As you might expect, she’s not big on this particular scientific consensus. I forgot about it, but then the other day saw this Tweet exchange between Dr. Orient and Ed Wiebe:
@edwiebe @AAPSonline I can’t vote against U.S, bureaucrats, much less UN. 31000 scientists voted against IPCC. http://t.co/fdJDgA5bu6
— Jane Orient, MD (@jorient) December 13, 2014
She’s referring to a project by AGW denialists that’s very much like the Perth Group when it comes to HIV/AIDS denialists and various lists of scientists who purported “dissent from Darwin.” It’s a common tactic used to try to give the appearance of an actual scientific controversy when in fact there isn’t one. Not surprisingly, similar efforts are taking place in the AGW community, and, also not particularly surprisingly, JPANDS has been linked to the Petition Project, which since 1998 has been trying to get signatures from scientists who don’t believe AGW is happening, while JPANDS itself has published some really bad papers on climate change. Later in the exchange, Dr. Orient Tweeted:
@edwiebe @AAPSonline You suggested voting on UN IPCC. Glad you agree that alleged consensus on AGW is not science.
— Jane Orient, MD (@jorient) December 13, 2014
This reminded me of an earlier Tweet to me:
@secularbloke @gorskon @AAPSonline Skepticism is the essence of science; consensus is its death
— Jane Orient, MD (@jorient) December 1, 2014
On the surface, Dr. Orient sounds so reasonable, doesn’t she? Isn’t it good for a scientist to be skeptical of everything, including existing scientific consensus? There’s just one problem. Dr. Orient completely misunderstands skepticism and science. Note her hostility at the very concept of that such a thing as a scientific consensus exists about anything, and you’ll be closer to seeing what her true agenda is.
It has often been written on this blog and elsewhere that the mark of a true crank is hatred of the scientific consensus, be it consensus regarding the theory of evolution; the science that says homeopathy is impossible; the scientific consensus among climate science that concludes that the earth is getting warmer due to human activity; various areas of science-based medicine; or the safety and efficacy of vaccines. Perhaps the most famous expression of distrust of a scientific consensus is the famous and widely quoted speech by Michael Crichton, in which he said:
Let’s be clear: the work of science has nothing whatever to do with consensus. Consensus is the business of politics. Science, on the contrary, requires only one investigator who happens to be right, which means that he or she has results that are verifiable by reference to the real world. In science consensus is irrelevant. What is relevant is reproducible results. The greatest scientists in history are great precisely because they broke with the consensus.
There is no such thing as consensus science. If it’s consensus, it isn’t science. If it’s science, it isn’t consensus. Period.
To which I (and many others) responded, “Bullshit!” (Seriously, anything less is not a strong enough rebuke.)
In fact science is all about coming to a consensus, but it’s about coming to a consensus based on data, experimentation, and evidence, a consensus that has reproducible results that are, as Crichton put it, verifiable by reference to the real world. After all, what is a scientific theory like the theory of evolution or Einstein’s theory of relativity but a statement of the current scientific consensus regarding a major scientific topic? What is peer review but quality control (making sure the scientific methodology is sound) coupled with testing new science against the current consensus to see where it fits in or where it exposes weaknesses? What is science but attempting to forge a consensus regarding theories and statements that most accurately describe the universe in a useful and predictable way?
Of course, questioning the consensus is often necessary in science. Indeed, it is critical to scientific advancement. However, there is a huge difference between questioning a current consensus and producing the data and experimental evidence to show that there is a real scientific reason to question it, and JAQing off about science. The latter, raising spurious or already answered questions about a scientific finding or theory one doesn’t like, belongs to the province of cranks and denialists, and it is what they are very good at. The problem is that they aren’t very good at realizing why their questions are not worthy of the attention that they think they are. A few years ago, a lovely example of this showed up in the Discovery Institute’s propaganda arm, its version of Age of Autism, so to speak, namely Evolution News and Views. In it, the Kent Heckenlively of the creationist set, the ever excitable Casey Luskin, penned a typical bit of silliness in which he asks the question, ‘When Is it Appropriate to Challenge the “Consensus”?‘
If Mr. Luskin had one clue about science, he could answer the question in two sentences and echo how scientists would answer the question: When you have an actual scientifically-valid reason, based on science, evidence, experimentation, and observational evidence, to think that the current scientific consensus about something is in error, then it is appropriate to challenge the scientific consensus. When you don’t, then it isn’t. Unfortunately, Casey didn’t; so he couldn’t. Instead, we were treated to a potpourri of pseudoscientific and denialist claptrap that was apparently based on an article in The American by Jay Richards entitled “When to Doubt a Scientific ‘Consensus’.” In the article, Richards postulates twelve “signs” that should lead one to doubt a scientific consensus, any scientific consensus—although he seemed most concerned with anthropogenic global warming in this particular article, while Luskin is, of course, concerned mostly with “intelligent design” creationism versus the hated (by Luskin) “Darwinism.” There’s just one problem. Not a single one of these “signs” had anything to do with a scientific argument. Richards started out with a reasonable enough introduction:
Anyone who has studied the history of science knows that scientists are not immune to the non-rational dynamics of the herd. Many false ideas enjoyed consensus opinion at one time. Indeed, the “power of the paradigm” often shapes the thinking of scientists so strongly that they become unable to accurately summarize, let alone evaluate, radical alternatives. Question the paradigm, and some respond with dogmatic fanaticism.
We shouldn’t, of course, forget the other side of the coin. There are always cranks and conspiracy theorists. No matter how well founded a scientific consensus, there’s someone somewhere–easily accessible online–that thinks it’s all hokum. Sometimes these folks turn out to be right. But often, they’re just cranks whose counsel is best disregarded.
So what’s a non-scientist citizen, without the time to study the scientific details, to do? How is the ordinary citizen to distinguish, as Andrew Coyne puts it, “between genuine authority and mere received wisdom? Conversely, how do we tell crankish imperviousness to evidence from legitimate skepticism?” Are we obligated to trust whatever we’re told is based on a scientific consensus unless we can study the science ourselves? When can you doubt a consensus? When should you doubt it?
Your best bet is to look at the process that produced, maintains, and communicates the ostensible consensus. I don’t know of any exhaustive list of signs of suspicion, but, using climate change as a test study, I propose this checklist as a rough-and-ready list of signs for when to consider doubting a scientific “consensus,” whatever the subject. One of these signs may be enough to give pause. If they start to pile up, then it’s wise to be suspicious.
So far, there isn’t much here to disagree with. Scientists are human beings; scientific fads come and go. Ditto physicians. Medical fads come and go too. Some scientific consensuses ultimately turn out to be wrong. Virtually all of them undergo significant revisions as new evidence comes in. Moreover, not all consensuses are created equal. Depending upon the field, the strength of any one scientific consensus can vary quite markedly compared to others, depending upon the science, the topic within that science, or even the subtopic within the topic. For example, the scientific consensus supporting the theory of evolution, particularly common descent, is exceedingly strong. Based on multiple lines of converging evidence from many different disciplines, evolution is one of the strongest of all scientific consensuses. Similarly, the consensus that natural selection is one major driving force behind much of evolution is nearly as strong.
However, as the discussion devolves into more detailed, specialized areas, inevitably the consensus weakens. Eventually, subsidiary areas of a discipline are reached where the consensus is weak or where there is no consensus. Often these questions are at the frontiers of the science and, because there is not yet a consensus, they are the most heavily-researched and hotly-contested areas of the science. Denialists often attack science at the very edges of a field as a proxy for attacking the much more strongly supported core theory. Creationists like Casey Luskin are actually notorious for this, jumping on new findings about, for example, “junk DNA,” whether it has a function, whether it is subject to natural selection, and, if so, how much, as a bit of logical prestidigitation to hide the fact that the core theory of evolution is supported by mountains of evidence and not in doubt by scientists. Cancer quacks like to latch onto the newest findings to try to make it seem as though “everything you thought you knew about cancer is wrong.”
It is also true that peer pressure and groupthink can make persuading scientists that a particular scientific consensus is in error can be a disturbingly slow and messy process at times. However, in the end eventually science almost always wins out. One example (summarized very well by our very own Kimball Atwood IV, MD) is the discovery that most duodenal ulcers are actually caused by a bacterium, H. pylori. Barry Marshall and Robin Warren first reported a curious finding of what they described as “unidentified curved bacilli on gastric epithelium in active chronic gastritis” (not an ulcer) in two letters to The Lancet, published on June 4, 1983. They reported that it wasn’t seen using traditional staining methods and suggested that they might be associated with gastritis. By 1992, multiple studies had been published establishing the causative role of H. pylori in peptic ulcer disease, and medical practice rapidly changed. That’s less than ten years, which, given how long it takes to organize and carry out clinical trials, is amazingly fast. Yet somehow a favorite denialist myth is that “dogmatic,” “close-minded” scientists refused to accept Marshall and Warren’s findings. It’s an example of a scientific consensus that deserved to be questioned, was questioned in the right way, and was overthrown.
In other words, it was nothing like Richard’s twelve “signs”:
     1.  When different claims get bundled together.
     2.  When ad hominem attacks against dissenters predominate.
     3.  When scientists are pressured to toe the party line.
     4.  When publishing and peer review in the discipline is cliquish.
     5.  When dissenting opinions are excluded from the relevant peer-reviewed literature not because of weak evidence or bad arguments but as part of a strategy to marginalize dissent.
     6.  When the actual peer-reviewed literature is misrepresented.
     7.  When consensus is declared hurriedly or before it even exists.
     8.  When the subject matter seems, by its nature, to resist consensus.
     9.  When “scientists say” or “science says” is a common locution.
     10.  When it is being used to justify dramatic political or economic policies.
     11.  When the “consensus” is maintained by an army of water-carrying journalists who defend it with uncritical and partisan zeal, and seem intent on helping certain scientists with their messaging rather than reporting on the field as objectively as possible.
     12.  When we keep being told that there’s a scientific consensus.
Oddly enough, most, if not all, of these warning signs apply to denialists and cranks. Richards appeared to have been engaging in a massive case of projection. I’m not going to examine each of the twelve “signs” in detail (that will be left as an exercise for the interested reader), but I will examine a few of the most egregious “signs.” For example, when it comes to ad hominem attacks, Richards writes:
Personal attacks are common in any dispute simply because we’re human. It’s easier to insult than to the follow the thread of an argument. And just because someone makes an ad hominem argument, it doesn’t mean that their conclusion is wrong. But when the personal attacks are the first out of the gate, and when they seem to be growing in intensity and frequency, don your skeptic’s cap and look more closely at the evidence.
Crank movements, of course, excel at the ad hominem attack. Creationists like Casey Luskin, for instance, spit the term “Darwinist” at evolutionary biologists and frequently try to link evolution (and thus its defenders) Nazi-ism and the Holocaust, eugenics, social Darwinism, and all manner of evils. Above all, evolutionists must be atheists, which to many creationists is the worst thing a person can be, given the vehemence of the invective.
Speaking of invective, one crank in particular, J.B. Handley, has made a special study of seeing just how nasty his attacks can be. Generation Rescue and its propaganda arm Age of Autism specialize in “venomous invective,” particularly against Paul Offit and anyone else who opposes its anti-vaccine agenda. After all, this is the same man who launched personal attacks on Steve Novella that can only be viewed as more than venomous. This is the same man whose misogynistic attacks on Amy Wallace, a journalist who wrote an excellent article on the anti-vaccine movement, made him infamous throughout the science-based blogosphere. This the same man whose blog posted a Photoshopped picture of Steve Novella, Amy Wallace, Paul Offit, and Trine Tsouderos sitting around the table for a Thanksgiving feast, the main course of which was a baby.
If we look at the “case study” used by Richards, AGW denialists also excel at the same tactics, painting scientists as hopelessly politically motivated, corrupt, and lying. They hack e-mails looking for dirt and try to embarrass scientists by posting them. They attack Al Gore, the most famous advocate of political action to mitigate the effects of AGW, as fat, stupid, and corrupt, as though discrediting Al Gore would discredit AGW. (Hint: It won’t.) The list goes on.
By Richards’ criteria, the vaccine, evolution and AGW denialists send up huge red flags, as a major component of their message consists of ad hominem attacks on scientists. As for “misrepresenting the actual peer-reviewed scientific literature,” what are The Discovery Institute, Age of Autism, NaturalNews.com, and every other denialist website or blog but veritable fonts of misrepresenting scientific literature? Hardly a week goes by, it seems, that I’m not trying to refute some bit of nonsense or other about a scientific study laid down by the anti-vaccine movement or some quack or other. If you count my not-so-secret other blog, actually, hardly a day goes by without my having to do that.
It’s also hard not to note a distinct feeling of repetition in Richards’ list. For example “When ‘scientists say’ or ‘science says’ is a common locution” and “When we keep being told that there’s a scientific consensus” are more or less the same thing. Perhaps the silliest part of his list is this:
A scientific consensus should be based on scientific evidence. But a consensus is not itself the evidence. And with really well-established scientific theories, you never hear about consensus. No one talks about the consensus that the planets orbit the sun, that the hydrogen molecule is lighter than the oxygen molecule, that salt is sodium chloride, that light travels about 186,000 miles per second in a vacuum, that bacteria sometimes cause illness, or that blood carries oxygen to our organs. The very fact that we hear so much about a consensus on catastrophic, human-induced climate change is perhaps enough by itself to justify suspicion.
Now there’s some serious misunderstanding there.
Richards apparently doesn’t know the difference between scientific theory and scientific fact. That salt is sodium chloride is a fact. That light travels 186,000 miles per second in a vacuum is a measurement and a fact. That blood carries oxygen to our organs is a fact. Of course, no one argues about them; they are well-settled facts, not theories. They are trivially obvious. Arguing about them would be as trivial as arguing about what I had for breakfast this morning or whether the above paragraph by Richards represents the essence of scientific ignorance. A theory is a higher level construct supported by facts, experimentation, and evidence.
Casey Luskin’s and Jay Richard’s tag-team effort demonstrated to me a profound ignorance of science—even an anti-scientific bent. They don’t like science because it either doesn’t support their political beliefs (Jay Richards and AGW) or their religious beliefs (Casey Luskin and evolution). Sure, scientists can at times be as petty as any human being. They are as prone to groupthink and ideology as any group of people can be. But the wonderful thing about being a scientist is that science is a process. Although it is an activity of people it does not depend on any group of people. Eventually, even when scientists go down wrong alleys or succumb to fads, science wins out. It is self-correcting. The process may not be as fast as we like. It may not be as linear as we like. In fact, sometimes it’s damned messy and frustrating. However, it is the best process we have for finding out how our universe works.
Oh, and for building a consensus about how the universe works. It’s perfectly acceptable to challenge such a consensus, but if you don’t have the goods in the form of evidence, experimentation, and data to show that the consensus is in serious error, there is no reason for scientists to take your challenge seriously. As Joshua Rosenau put it:
But moving past those trivialities, Casey and Jay’s underlying point is catastrophically wrong. As John Ziman points out in Reliable Knowledge: “the goal of science is a consensus of rational opinion over the widest possible field” (emphasis original). The beauty of science is precisely that it is rooted in our shared reality, and as such it is subject to the formation of consensus on which new work can build.
Yep, that’s about right. Contrary to what Dr. Orient claims, consensus isn’t the death of science; rather, it’s what science strives to reach in all its argumentative, complicated, and, yes, messy glory. Indeed, I can’t resist asking again: What is a scientific theory but a scientific consensus about how one aspect of how the world works?
It should thus come as no surprise that the AAPS is not so fond of evidence-based medicine (EBM), in particular the treatment guidelines that flow from EBM. Indeed, JPANDS has published several articles with titles such as “Evidence-based Guidelines: Not Recommended“, “The Effect of Peer Review on Progress: Looking Back on 50 Years in Science” (featuring another scornful dismissal of the “herd instinct” and “conformity” and a fair amount of exaggeration of how much scientific progress is due to “violent confrontation” of old paradigms and how much is due to the slow accumulation of knowledge), and editorials attacking evidence-based medicine. To the AAPS, evidence-based guidelines are unacceptable limits on the autonomy of physicians.
I’d like to finish with an observation. Like Harriet Hall, Steve Novella, and Kimball Atwood, I’m a fellow of the Center for Inquiry, and I was happy to sign a statement from the Committee for Skeptical Inquiry entitled “Deniers are not Skeptics“. It’s a statement that asks the media to “please stop using the word ‘skeptic’ to describe deniers,” much as I’ve on occasion railed on descriptions of antivaccinationists as “vaccine skeptics” or cancer quacks as “chemotherapy” skeptics, or activists like Vani Hari (a.k.a. “The Food Babe”) as “skeptics” of the food industry. Here’s why:
As Fellows of the Committee for Skeptical Inquiry, we are concerned that the words “skeptic” and “denier” have been conflated by the popular media. Proper skepticism promotes scientific inquiry, critical investigation, and the use of reason in examining controversial and extraordinary claims. It is foundational to the scientific method. Denial, on the other hand, is the a priori rejection of ideas without objective consideration.
Real skepticism is summed up by a quote popularized by Carl Sagan, “Extraordinary claims require extraordinary evidence.” Inhofe’s belief that global warming is “the greatest hoax ever perpetrated on the American people” is an extraordinary claim indeed. He has never been able to provide evidence for this vast alleged conspiracy. That alone should disqualify him from using the title “skeptic.”
The same can be said of all manner of medical cranks, including antivaccinationists, cancer quacks, and the like, all of whom use the same techniques of false argumentation to attack medical consensuses they don’t like, such as the one that concludes vaccines are safe and effective. Unfortunately, physicians are no different. If there’s one thing that identifies a scientific crank or a medical quack, it’s hatred of the scientific consensus. I’m not just talking about questioning a consensus, which scientists and physicians do every day, but attacking the very concept of scientific consensus. Michael Crichton taught me that inadvertently, and Dr. Orient continues to do so. To overthrow an existing scientific consensus requires more than they can give: Lots of evidence and valid scientific reasons to do so.

Public Attitude Toward Science
New Pew research findings show confidence in science remains high, but there are some important caveats.
Steven Novella on February 19, 2020
Link


A newly released Pew Research Center survey has some interesting details about American’s attitudes toward science and scientists. The data has implications for our efforts here at SBM, and I think it contains some encouraging news. It deals with both public attitudes, as well as demographic factors that influence those attitudes, such as education level, knowledge, and political affiliation.
The first item in the report deals with vaccination – the question is, “The benefits of the measles vaccine outweighs the risks.” Overall 88% of responders agreed that the benefits outweigh the risks. This is not too bad, but is a bit low if our goal is herd immunity, which for many vaccines requires something closer to 90-95% compliance. Confidence in the measles vaccine increased with education and income level, and was 93% among college grads and post-grads, and 95% among those in the upper income. This suggests that education can have an effect, and pushes acceptance into the herd immunity range.
Further, political affiliation had essentially zero effect on this question. This finding is in line with previous surveys which have also found that vaccine attitudes are not partisan. In fact, prior research has shown that those with anti-vaccine attitudes display a “super Dunning Kruger” effect – they know the least about immunity and the causes of autism while thinking they know the most, even more than experts. The problem is that they are filled with misinformation that gives them the powerful illusion of knowledge.
At least in this case efforts at debunking misinformation and spreading accurate information should have a positive effect, especially when combined with promotion of critical thinking skills (such as an understanding of the pitfalls of conspiracy thinking). Vaccine denial may be one of the exceptions to the otherwise general inadequacy of the knowledge deficit model of science communication. The other known exception is opposition to genetically modified organisms, which again displays a super DK phenomenon where GMO opponents have the least knowledge of genetics but think they have the highest.
The rest of the Pew survey focused on general attitudes toward science and scientists, with some mixed news. The percentage of Americans who report a fair or great deal of confidence in medical researchers was the highest, at 87%, with scientists in general at 86%. These numbers are also up from 2016, at 84% and 76% respectively. This might suggest that in a politically chaotic time the public turns more to institutions they think are objective or at least non-partisan.
These numbers are also higher than any other institution – military (82%), religious leaders (57%), news media (47%), business leaders (46%),, and at the bottom elected officials (35%). This generally high level of confidence in the institutions of science is certainly helpful to our efforts to increase scientific literacy and improve understanding of medical science in particular.
But politics does complicate this overall confidence in science. Only 60% of Americans said that scientists should play an active role in policy debates about scientific issues, while 39% agreed that scientists should stick to establishing scientific facts and stay out of policy. These numbers were also heavily influenced by political affiliation, with 73% of Democrats and only 43% of Republicans favoring scientist involvement in policy discussions. This suggests that for many people support of the institutions of science is thin, or restricted to the abstract. But they don’t want scientists “meddling” in policy – even on scientific issues.
Another question puts this into further context. “The scientific method generally produces accurate conclusions,” vs “Can be used to produce any conclusion the researcher wants.” This is the problem with such forced-choice false dichotomy surveys – I think that both of these statements are simultaneously true, and I am not even sure how I would have answered this question. The problem is, there is simply too much nuance in this topic, and of course regular readers know that we spend a great deal of time at SBM exploring that nuance.
I would summarize my own attitude as – the scientific method, when properly applied, can produce generally accurate conclusions, eventually. However, this only works in the aggregate, over multiple studies and multiple researchers, with replication and other factors being carefully considered. For individual studies the scientific method can be twisted to produce any result the researcher wants. We have discussed at length researcher degrees of freedom, P-hacking, and publication bias, for example. The scientific method is neither a magic wand nor a scam – it works, but only if properly and carefully employed.
What all these Pew findings suggest is that respect for science and scientific institutions is high, but many people reserve some skepticism about the findings of science and the role of scientists in public policy. This skepticism seems to be dramatically influenced by political affiliation. In fact, on this question about the accuracy of the scientific method, the researchers broke it down by scientific knowledge and party affiliation. Among Democrats, scientific knowledge had a significant effect, with confidence ranging from 86 to 67 to 52% for high, medium, and low scientific knowledge respectively. For Republicans the numbers were 59, 52, and 51% respectively.
Essentially being a self-identified Republican mostly erased the positive effect that scientific knowledge has on confidence in the scientific method. This likely has a complex set of causes, but one issue stands out – that of climate change. What previous data shows is that, for Democrats, increased scientific knowledge positively correlates with acceptance of the consensus on climate change (and is generally high). For Republicans, however, there is an inverse relationship – increased scientific knowledge correlates with lower acceptance of the science of climate change. For this issue, there is no knowledge-deficit problem. Political affiliation is the dominant factor.
What does all of this mean for our efforts at SBM? I think generally there is some good news – confidence in science and scientists is high. In some medical areas, like vaccines and genetics, there is a knowledge deficit problem that can be fixed, and there is also a misinformation problem that can also be fixed. The data also suggests that we should continue our efforts to explore the relationship between science and medicine.
There is caution in the numbers as well. Politics still has the power to trump science and facts. Further, this and other data suggest that when science is attacked over a single political issue, it reduces confidence in science in general. This is something that we at SBM have been warning about for a long time. We often file this under the indirect harm of alternative and unscientific medicine. Attacks on the institutions and methods of science over one issue tend to depress confidence in science and practitioners in general, which makes the public more vulnerable to charlatans and snake-oil peddlers. When you claim that magic water works, even if you are relying on placebo effects, that has negative ripple effects in society.

Why People Deny Science
A new study further shows that people are inconsistent in how they justify their beliefs.
Steven Novella on November 14, 2018
Link

Promoting science-based medicine involves many components – discussing the often-complex relationship between research and best practice, examining specific claims, promoting science-based regulations, and addressing the many ways in which people try to undermine the scientific basis of medicine.
To those naïve to the challenges we face, at first the concept of science-based medicine seems obvious. Well of course you’re going to base medicine on science, what are the other options? At its core the idea is simple: medical practice should be informed by the best evidence we have available. In practice, this is complicated, because there are many ways to evaluate the evidence.
Further, people find many ways to deny the science, either a specific scientific conclusion, or science itself. We hear these various excuses all the time – there are “other” ways of knowing, we don’t need science to know what works, I am the evidence, etc. Sometimes people respect science, but just get it wrong. They might misunderstand the placebo effect, overestimate the significance of studies, not understand the nature of p-hacking, or fail to realize the potential for self deception in less-than-rigorous studies.
Then there are those who will dismiss entire swathes of science out of hand. This is commonly done through an appeal to conspiracy theories, such as references to “Big Pharma”, or the notion that doctors lie to make money, that the system is broken and cannot be trusted, or even that science itself is broken.
While there are differences in cognitive style, the research and common experience suggest that most people engage in many of these tactics at various times. These cognitive flaws and biases are not something that other people do, they are things that we all do, unless we vigilantly and carefully guard against them.
A new study further supports this conclusion. Emilio Lobato and Corinne Zimmerman did a study of 244 students and faculty members, asking them their opinions on GMOs, climate change, vaccines, and evolution. They also asked them open-ended questions about what would change their mind on these topics. What they found was that people used inconsistent reasoning across the four topics.
Some patterns did emerge, in line with previous research, showing a strong correlation between an analytical style of thinking and acceptance of the scientific consensus. There was also a positive correlation with a liberal ideology, and a negative correlation with religious belief and conspiracy ideation. Otherwise, the study showed a “consistent inconsistency”.
Subjects were asked to justify their rejection of the scientific consensus. In 33% of cases, one third, subjects simply restated their position, essentially giving no justification. In 34% of cases the subjects did cite evidence. In 20% of cases the subjects referenced their cultural or religious identity. So only about a third of the time did subjects reference evidence as the justification for their belief. This does not mean their belief is based on evidence – only that they justify the belief that way.
We know from other research that people will sometimes come to a conclusion for emotional reasons (identity, ideology) and then rationalize that belief, citing evidence or arguments that were not the real reason for their belief in the first place. They will also resist changing their position, even in the face of solid evidence, if their belief is emotionally held.
There are many studies showing that people will engage ad hoc in motivated reasoning, meaning that the conclusion comes first, and reasoning is used to justify the conclusion rather than determine the conclusion. There is inconsistent evidence for a possible backfire effect – which means not only rejecting evidence which contradicts a held belief, but strengthening that belief in the face of contradictory evidence.
The new study also shows evidence of motivated reasoning. Subjects would use various methods to deny the science, shifting from topic to topic. Subjects might cite evidence for one topic, then personal belief for another, then give no justification for a third. Only 11% of subjects cited evidence to justify their position on all four topics.
When subjects were asked what would potentially change their mind on a topic, 45% of subjects stated for at least one topic that nothing would change their mind, and 17% said this of more than one topic. On the positive side, 80% of subjects said that evidence would change their mind on at least one topic, but not a single subject said this about all four topics.
What all of this suggests is that people do not usually engage in metacognition – thinking about their own thinking. They may have a cognitive style that they tend to use, but otherwise they engage in whatever type of reasoning serves their purpose on any particular topic. They might vigorously defend the consensus of scientific opinion on one topic, then reject it on another citing a vague conspiracy, and dismiss it on a third without any real justification or by appealing to fallacious logic.
To counter this we cannot just teach science or explain what the evidence says. We need to teach critical thinking skills – which is metacognition. Critical thinking includes an understanding of all the various ways our thinking can go wrong. Just as important, however, is that critical thinking involves stepping back from our own cognitive process to examine it objectively, to make a sincere effort to consistently apply valid logic and the same fair and objective criteria for evidence.
Doing this is really hard, because people are very good at motivated reasoning. We are extremely creative at inventing reasons to reject logic and evidence we don’t like, and those invented reasons can create a powerful illusion of being correct – so much so that almost half of people feel comfortable saying that no new evidence could even theoretically change their mind on a topic.
We are also not doing this alone – we are social creatures and have a robust social network through which we spread ideas. In many cases motivated reasoning comes prepackaged. All the work of inventing creative reasons, cherry picking evidence, twisting logic, attacking scientists and institutions, and making emotional appeals has already been done. It’s even been market-tested, tweaked, and improved. The result is slick (and often well-funded) science-denial that takes real dedication to unpack.
The good news is that critical thinking skills are broadly applicable. That’s why I often encourage teaching critical thinking around topics that are less emotional (for the target audience), and then slowly encouraging the application of those critical thinking skills to more and more emotionally held beliefs. This is actually a life-long process, and it’s never done.
All we can hope to do is move the needle slowly in the direction of increased scientific literacy and critical thinking. That is ultimately the only way to promote science-based medicine.

Defending CAM with Bad Logic and Bad Data
Steven Novella on December 14, 2011
Link
At SBM our mission is to promote the highest standards of science in medicine, and to explore exactly what that means, both in the specific and the general. We do spend a lot of space criticizing so-called CAM (complementary and alternative medicine) because it represents a semi-organized attempt to reduce or even eliminate the science-based standard of care, and to sow confusion rather than clarity as to how science works and what the findings of medical science are.
CAM proponents tend to use the same bad arguments over and over again. They have no choice (other than deciding not to be CAM proponents) – if a treatment were backed by solid logic and evidence it would not be CAM, it would just be medicine. As SBM’s fourth year comes to a close I thought I would round up the most common bad arguments that CAM proponents put forward to defend their position. Like creationists, pointing out the errors in their facts and logic will not stop them from continuing to use these arguments. But this lack of imagination on their part makes it somewhat easy to counter their arguments, since the same ones will come up again and again.
The argument from antiquity
Our SBM colleagues in Australia have been critical of the incorporation of unscientific methods into academia. In defense of this practice:
Professor Iain Graham from Southern Cross University’s School of Health yesterday defended his university, saying the use of alternative therapies, such as homeopathy, can be traced as far back as ancient Greece.
This is a common claim – that some CAM modalities have been around for centuries, or even thousands of years, and so they must work. I am not sure if professor Graham intended to state that homeopathy can be traced back to ancient Greece, perhaps he just meant that some CAM therapies can, and chose homeopathy as a bad example. For the record, homeopathy was invented by Samuel Hahnemann about 200 years ago.
But I wonder what CAM modalities he had in mind. Chiropractic? About 100 years. Therapeutic touch? A few decades. Acupuncture is a complex question, but what passes for acupuncture today is less than 100 years old. Perhaps he was thinking about blood letting or trepanation.
It is true, however, that some basic concepts, like the notion of a life energy, can trace it roots to ancient Greece, and other ancient cultures. However, such notions are pre-scientific nonsense. Scientists abandoned the notion of life energy over a century ago because there was no evidence that such a force exists (and there still isn’t) and after figuring out all the basic processes of life there was essentially nothing left for the alleged life force to do.
For some reason, however, professor Graham believes that antiquity in science is a virtue – the “argument from antiquity” logical fallacy. The unstated assumption is that if an idea has survived for hundreds or thousands of years it must be legitimate. This is demonstrably false. Galenic medicine (blood letting, purging, etc. based on the notion of the four humours) survived for thousands of years, and yet it was based on complete and utter primitive nonsense. In fact its tendrils still exist – there is still blood letting, cupping (which is just another form of blood letting), and similar practices going on in the world. It was replaced in the West because of the advent of science in medicine – a trend that Graham and other CAM proponents apparently want to reverse.
The argument from popularity
Graham also states in the same article:
“Eighty per cent of Australians seek alternative therapies,” Prof Graham said.
“Obviously orthodox medicine is not working for everyone,” he said.
The argument from popularity is almost ubiquitous in CAM apologetics. We are constantly being told that CAM is popular and that its popularity is growing. This argument is used to justify incorporation of CAM into academia, spending research funds on CAM, and licensing CAM practitioners.
As with the argument from antiquity, the facts are often fudged. For example, I highly doubt that the 80% figure quoted for Australia is correct. Most such figures are highly inflated by including all sorts of practices in the CAM category, like exercise, eating organic food, and sometimes prayer is included. US surveys show the percentage of CAM use is around 1/3, but this is mostly things like massage and chiropractic manipulations. Homeopathy is around 3-4%, and acupuncture 6-7%. In fact, only manipulation and massage were in the double digits.
This is all marketing deception – create a false category (CAM), pad it out with commonly used methods, and then claim that the extreme fringes are therefore getting more popular. I don’t know how Graham got to 80% (I doubt such methods are that much more popular in Australia) but it is close to one survey from 2007 that found that 69% of Australians used one of the 17 most popular forms of CAM in the last year. However, they included in their list: martial arts, yoga, massage, meditation, and taking multivitamins. I am not sure what taking multivitamins says about the popularity of homeopathy, but apparently professor Graham thinks that is significant.
In any case – I will grant that CAM as a marketing concept has been somewhat successful, and even that it has gained popularity recently (although not as much as advocates would have you think). That is entirely irrelevant, however, to the question of whether or not any particular CAM modality is science-based and appropriate for a university curriculum.
Universities are supposed to be thought-leaders, to have intellectual standards that rise above the mere notion of popularity. They are supposed to uphold academic standards of scholarship, and in scientific disciplines high standards in science. The medical profession should be science-based and have high professional standards – not cater to the latest fad. It is therefore very odd and disturbing to defend a university policy or professional conduct based upon popularity. Should we allow surveys of public opinion to determine whether or not we teach creationism or astrology in our universities?
False choice
The second part of Graham’s quote was:
“Obviously orthodox medicine is not working for everyone,” he said.
This is a common assumption, even among CAM critics. If people are choosing to use CAM then they must be dissatisfied with mainstream medicine. This is a way of providing a false choice – if there are deficiencies in science-based medicine, then CAM is an appropriate “alternative.” Again – this is a similar strategy to the creationists, whose main style of argument is to attempt to poke holes in evolutionary theory, then propose creationism as the only alternative, without ever providing actual evidence for creation. CAM proponents therefore spend a lot of time criticizing mainstream medicine, and then present CAM as the only alternative.
The claim that people are using CAM because they are not satisfied with science-based medicine, however, is not based upon any evidence. In fact, what evidence we have strongly suggests this is wrong. Surveys show that people use CAM because they want to increase their options, or because they are philosophically aligned with the marketing image created for CAM (holistic, etc.). CAM users, however, do not turn to CAM because they are dissatisfied with their doctors or mainstream medicine. In fact in one survey CAM users were more satisfied with their primary care doctors.
No one claims that mainstream medicine is perfect, or that science-based medicine has all the answers. As with science in general – medicine is a flawed human endeavor. The rational response to the deficiencies in medicine, however, is to strive to improve our knowledge and practice, not to turn to an alternative simply because it’s there. CAM therapies need to stand on their own merits (which they can’t – that’s why they are not accepted as science-based). Proponents, however, promote CAM therapies as if they are the default choice when SBM is imperfect.
Tu quoque
A similar strategy to the false choice is to justify the failings of CAM by pointing out the failings in mainstream medicine – the tu quoque logical fallacy.
In the comments to the above article commenter “shotinfo” (which, I am told, is a common pseudonym for Meryl Dorey, who heads an anti-vaccine organization in Australia) wrote:
According to articles published in both the British Medical Journal and the New England Journal of Medicine, between 85% and 90% of all mainstream medical drugs and procedures have never been scientifically proven to either work or be effective.
I love the fake references – of course no such articles in BMJ or NEJM exist, and no references were offered, but it certainly makes the fake factoid sound legitimate. I have already discussed this issue in detail. The bottom line is that surveys of medical practice find that about 78% of them are reasonably evidence-based. Yet again we have a factual misstatement used to support a logical fallacy (tu quoque). All medical practice should strive to be more science and evidence-based. Pointing out the deficiencies in one discipline does not justify deficiencies in another.
The commenter, however, also misses the real point of criticism of CAM. Mainstream medicine is based upon a culture and institution of science, and a science-based standard of care. The execution of this standard is flawed, but the principle is clear. CAM is not based on a science-based standard. It, in fact, seeks to subvert and even remove the science-based standard of care. And CAM proponents live in a culture of pseudoscience, not legitimate science.
To the extent that the science-base of modern medicine is not adequate the proper response is to improve the science (a never ending task), not to use methods that are even less science-based. CAM is largely devoid of science. CAM proponents, to borrow an excellent turn of phrase, use science as a drunk uses a lamppost – for support rather than illumination. Tom Harkin gives us the best evidence for this when he stated (talking about the NCCAM):
One of the purposes of this center was to investigate and validate alternative approaches. Quite frankly, I must say publicly that it has fallen short. It think quite frankly that in this center and in the office previously before it, most of its focus has been on disproving things rather than seeking out and approving.
The purpose of science is to validate, not disprove, according to Harkin. This, of course, is the opposite of the truth, but reveals a key logical failing of the CAM community.
Conclusion
There are many more examples of poor logic and dubious facts common to the CAM community (“what’s the harm,” for example), but the above arguments are the most common in my experience. After discussing and debating this issue for years it is clear to me that the above arguments will not go away simply because we point out that they are false and logically invalid. So we will have to keep pointing it out, over and over again.

Unfalsifiable Beliefs
Steven Novella on January 21, 2015
Link

As we search for a logo for SBM or the SfSBM, Mark Crislip has been a strong advocate of using an image of Sisyphus, endlessly pushing a boulder up a hill only to have it roll back down again. It’s a bit too self-defeating to be enthusiastic about that suggestion, but it does reflect a common feeling among all of us here at SBM – promoting science can be a frustrating endeavor.
Our frustration reflects a broader phenomenon, that it is difficult to persuade people with facts and logic alone. People tend to prefer narrative, ideology, and emotion to facts. The high degree of scientific illiteracy in the culture presents another barrier.
In recent years psychologists have demonstrated experimentally what we have come to understand through personal experience, that people engage in a host of cognitive defense mechanisms to protect their beliefs from the facts. We jealously guard our world view and are endlessly creative in shielding it from refutation.
A recent series of experiments published by Friesen, Campbell, and Kay in the Journal of Personality and Social Psychology demonstrates that one strategy commonly used to protect our beliefs is to render them unfalsifiable, or at least incorporate unfalsifiable elements. 
What they demonstrate is that people will shift from factual to unfalsifiable justifications for their beliefs as needed, depending on the facts with which they are confronted. For example, critics of same-sex marriage were confronted with statistics showing that children raised by same-sex couples either did well or were troubled. When given facts that supported their belief, subjects defended their beliefs on factual grounds. When given facts that contradicted their beliefs, they resorted to moral opinion.
The authors also found that unfalsifiable elements (subjective opinion, moral judgments, etc.) were used to more-aggressively criticize opponents. This strategy was therefore used both offensively and defensively.
This research is in line with previous research, summarized here by Chris Mooney, demonstrating that when people are confronted with facts that contradict strongly-held political opinions, they will tend to strengthen their original opinion. Facts just motivate people to engage in motivated reasoning to defend their position, strengthening it.
People engage in a number of cognitive strategies to deny evidence and maintain strongly-held beliefs. We encounter all of them here at SBM, which is why some people still believe, for example, that magic water can cure anything. As the recent research shows, one such strategy is to make one’s belief partly or entirely unfalsifiable. The examples used by the researchers led to the “moral opinion” approach, but there are others.
In the context of medicine we often encounter the “you can’t test my treatment” defense. Mystical energies are too subtle to be detected by science, for example. When I was on the Dr. Oz show he defended his faith in acupuncture (despite the negative scientific evidence) by stating that Western science does not know how to test this Eastern wisdom and that it was arrogant to think that it could.
This represents a move away from science as a method for knowing which treatments are safe and effective, something also shown by the research. When scientific evidence does not support your world view, this may lead to rejecting science itself.
Another method is to reject only the science that does not agree with you. You can do this by cherry picking the science you like. You can also come up with reasons to dismiss scientific evidence that contradicts your beliefs. In its simplest form this can be, “my experts are better than your experts.” You can always find a crank somewhere, or even a legitimate expert who just holds a minority or contrarian position.
Yet another method is to play the conspiracy/shill/”Big” card. Any scientist or journalist presenting an analysis or fact that goes against the preferred belief is labeled a shill for Big Whatever. If necessary the shill can be part of a massive conspiracy to hide the truth from the public. This is often presented as a cynical and simplistic “follow the money” argument – there is money in X therefore you cannot trust the powers that be.
If you read the comments for articles on medical topics, especially those addressing controversial topics, you will see comments dismissing scientific evidence with broad brushstrokes that simply refer in the abstract to the existence of Big Pharma, shills, the malfeasance of the government, and the corrupting influence of profit. There are also many articles and books by proponents of unscientific belief systems that essentially do the same thing, but in a more sophisticated form, giving details to the broad brush strokes, but still amounting to little more than a dismissal of inconvenient scientific evidence.
Conclusion
The grand social media experiment and formal psychological studies are showing the profound human tendency to prefer existing belief systems over being accurate and correct. Most frustrating is the tendency to engage in motivated reasoning to defend a position rather than alter one’s position to best accommodate existing scientific facts.
The specific mechanisms that people use to maintain their desired beliefs include: incorporating unfalsifiable elements such as moral opinions, subjective judgments, or stating that the phenomenon is not amenable to scientific investigation; dismissing evidence by attacking the messenger as being a shill; invoking conspiracies of “Big” whatever or government malfeasance; cherry picking evidence or experts; naked cynicism; or denying the role of science itself in addressing such issues.
The authors of the recent paper conclude that:
…in a world where beliefs and ideas are becoming more easily testable by data, unfalsifiability might be an attractive aspect to include in one’s belief systems, and how unfalsifiability may contribute to polarization, intractability, and the marginalization of science in public discourse.
This seems like a reasonable concern, and may be the unanticipated dark side of the vast increase in access to information afforded by the internet. In the past simple ignorance was enough to shield one’s belief system from refutation. Up until 20 years ago, if I was engaged in a discussion with a believer in homeopathy, for example, I could state that the evidence shows homeopathy does not work, and they could state the evidence shows it does work. Unless one of us was walking around with review articles in our pocket, that would be the end of it.
Today if you state an incorrect fact, it is highly likely that someone will provide one or more links to references that refute the incorrect fact, or even shove a smartphone in your face with the correct information. This encourages the development of skills that can be used to dismiss facts and the legitimacy of a specific science or science in general. Conspiracy theories, witch hunts, and sophisticated nonsense is therefore also on the rise to counter the threat that the ready availability of facts presents to belief systems. The ironic result is that access to facts may have a polarizing effect, rather than resolving differences.
One possible solution is to teach critical thinking skills to help more individuals transcend this evolved tendency to dig in one’s ideological heels. This is a long and difficult process, of course, but we will continue to push that boulder up the hill.

Fighting Against Evidence
Steven Novella on January 29, 2014
Link
For the past 17 years Edge magazine has put an interesting question to a group of people they consider to be smart public intellectuals. This year’s question is: What Scientific Idea is Ready for Retirement? Several of the answers display, in my opinion, a hostility toward science itself. Two in particular aim their sights at science in medicine, the first by Dean Ornish, who takes issue with large randomized controlled clinical trials, and the second by Gary Klein, who has a beef with evidence-based medicine.
These responses do not come out of nowhere. The “alternative medicine” meme that has taken hold in the last few decades (a triumph of slick marketing over reason) is all about creating a double standard. There is regular medicine which needs to justify itself with rigorous science, and then there is alternative medicine, where the rules of evidence bend to the needs of the guru or snake oil salesperson.
We have been hearing arguments from alternative medicine proponents for years now for why the strict rules of science need to be relaxed or expanded. Andrew Weil has advocated for the use of “uncontrolled clinical observations,” (also known as anecdotes). David Katz advocates for a “more fluid concept of evidence.” Dr. Oz went as far as advocating outright medical relativism, saying. “You find the arguments that support your data, and it’s my fact versus your fact.”
Dean Ornish
I can now add Dean Ornish to the list of gurus who want to change the rules of evidence to suit their needs. He is the founder and president of the Preventive Medicine Research Institute, and a long-time advocate of alternative medicine.
His answer to the question of what concept needs to be ditched from science is the large randomized controlled trial. He writes:
It is a commonly held but erroneous belief that a larger study is always more rigorous or definitive than a smaller one, and a randomized controlled trial is always the gold standard . However, there is a growing awareness that size does not always matter and a randomized controlled trial may introduce its own biases. We need more creative experimental designs.
He then launches into a discussion of the potential weakness and biases inherent in doing large trials. His specific observations are mostly valid, it is the conclusions he draws that are suspect.
Ornish is partly committing the Nirvana fallacy, or making the perfect the enemy of the good. No scientific study is perfect. Large clinical trials are very difficult to run, and compromises are often made for practical reasons. There are limited resources, and when studying people you have to consider the inconvenience to the study subjects.
The goal is to design the best study possible, not necessarily the perfect study. Further, multiple studies are often necessary, making different compromises, so that the strengths and weakness of the various trials will complement each other. In the end we grind our way slowly to a reliable answer.
It is easy, however, to point at the limitations and conclude that science is hopeless (the nihilistic approach) or that anything goes (the alternative medicine approach).
Ornish is claiming that smaller studies can be more rigorous because you can dedicate more time and resources to each subject. Really large trials compromise by spending fewer resources on each subject.
But, Ornish glosses over the fact that with smaller studies you sacrifice statistical power. It is odd to argue that smaller studies are better, or that we should abandon the large clinical trial. A more reasonable conclusion is that, with each trial, we make the compromises that make the most sense.
In general studies are powered just enough to be able to demonstrate the anticipated or likely effect size. Overpowering a study is a bad idea, but so is underpowering a study.
Further, small detailed studies can be complementary to larger simpler trials. So Ornish is also making a false choice fallacy.
Ornish’s odd recommendations perhaps make sense in light of the example he presents:
That’s just what happened in the Women’s Health Initiative study, which followed nearly 49,000 middle-aged women for more than eight years.
However, the experimental group participants did not reduce their dietary fat as recommended—over 29 percent of their diet was comprised of fat, not the study’s goal of less than 20 percent.
It seems he is whining a bit about a large clinical trial that did not show the results he liked. It’s OK to point out the weaknesses in the trial (although I think Ornish protests too much). It’s a radical overreaction to recommend ditching large clinical trials.
Gary Klein
It’s rare to see a logical fallacy stated so overtly. Klein could not have crafted a better example of the Nirvana fallacy if he tried:
But we should only trust EBM if the science behind best practices is infallible and comprehensive, and that’s certainly not the case. Medical science is not infallible. Practitioners shouldn’t believe a published study just because it meets the criteria of randomized controlled trial design. Too many of these studies cannot be replicated. Sometimes the researcher got lucky and the experiments that failed to replicate the finding never got published or even submitted to a journal (the so-called publication bias). In rare cases the researcher has faked the results. Even when the results can be replicated they shouldn’t automatically be believed—conditions may have been set up in a way that misses the phenomenon of interest so a negative finding doesn’t necessarily rule out an effect.
Really – unless science is infallible and comprehensive, we should ditch it? Unless we have perfect knowledge of everything we should behave as if we know nothing?
This attitude is not new. It is common in the alternative world. It just usually isn’t stated so boldly.
Again, Klein points out legitimate problems with the institution of science in general, and evidence-based medicine in particular. Yes – there are biases, there are publication issues and failure to replicate. We spend a great deal of time on SBM pointing out and discussing all the various challenges to rigorous science.
Klein and others, however, want to throw the baby out with the bathwater – to ditch scientific evidence, rather than work toward improving it. All of the problems with science in medicine have potential solutions, and we are making progress.
Klein also falls for a very common myth about EBM:
EBM formulates best practices for general populations but practitioners treat individuals, and need to take individual differences into account.
Here he clearly demonstrates that he is not familiar with EBM (and therefore is not in a position to recommend its demise). EBM absolutely recognizes that evidence needs to be applied to individual patients, and that practitioners need to combine the best evidence with their own clinical experience and judgments. This is nothing but a false accusation based upon ignorance of EBM.
Klein goes further, saying that advances in surgical techniques do not need placebo controlled trials, therefore we don’t really need placebo-controlled studies.
He then concludes:
Worse, reliance on EBM can impede scientific progress. If hospitals and insurance companies mandate EBM, backed up by the threat of lawsuits if adverse outcomes are accompanied by any departure from best practices, physicians will become reluctant to try alternative treatment strategies that have not yet been evaluated using randomized controlled trials. Scientific advancement can become stifled if front-line physicians, who blend medical expertise with respect for research, are prevented from exploration and are discouraged from making discoveries.
This is a profoundly naïve position. Preventing practitioners from essentially experimenting in an uncontrolled way on their patients is a good thing. Best practices and the standard of care exist for a reason – and they are not only based upon the best evidence, but also expert analysis and experience.
Adherence to best practices strongly correlates with better outcomes. Over-reliance on experience and judgement in deciding on treatments can be counterproductive.
Further, experimenting needs to be done within a strict ethical and scientific framework. You cannot, for example, ditch the standard of care in order to go exploring.
Conclusion
Doing clinical science is complex and difficult. The institutions of science are also imperfect and need continuous revisions.
There is a disturbing tendency to point out the challenges of rigorous science as a means of arguing for the abandonment or loosening of the rules of science and evidence.
This very approach is a logical fallacy, however. Limitations with science do not imply that anything works or is better. Further, science may have challenges but it is still the best game in town. Its problems are not insurmountable – they can be improved or fixed.
For example, journals can give more space to publishing replications, human trials should be increasingly registered so that negative studies cannot be hidden, and we need to lower the “publish or perish” pressure in academia to help encourage fewer more rigorous trials and reduce the flood of preliminary studies that are unreliable.
There are plenty of thoughtful solutions. Abandoning rigorous evidence is not one of them.

The Skeptical Clinician
Steven Novella on April 18, 2012
Link
All scientists should be skeptics. Serious problems arise when a less-than-skeptical approach is taking to the task of discovery. Typically the result is flawed science, and for those significantly lacking in skepticism this can descend to pseudoscience and crankery. With the applied sciences, such as the clinical sciences of medicine and mental therapy, there are potentially immediate and practical implications as well.
Clinical decision making is not easy, and is subject to a wide range of fallacies and cognitive pitfalls. Clinicians can make the kinds of mental errors that we all make in our everyday lives, but with serious implications to the health of their patients. It is therefore especially important for clinicians to understand these pitfalls and avoid them – in other words, to be skeptics.
It is best to understand the clinical interaction as an investigation, at least in part. When evaluating a new patient, for example, there is a standard format to the “history of present illness,” past medical history, and the exam. But within this format the clinician is engaged in a scientific investigation, of sorts. Right from the beginning, when their patient tells them what problem they are having, they should be generating hypotheses. Most of the history taking will actually be geared toward testing those diagnostic hypotheses.
For example, if a patient is being seen with a headache they may relate that the pain is most severe in the front of the head and extends down into the face. This could be a migraine headache, or a sinus headache, or less likely due to an underlying serious pathology. The physician may then ask if the patient has nasal congestion – this is not a random or screening question, but is designed specifically to test the hypothesis that the headache may be a sinus headache. During the exam the physician may also apply pressure to the sinuses to see if they are sensitive, and perhaps even this reproduces the patient’s headache.
This seems rather straightforward, but actually all of the potential pitfalls of scientific investigation are in play. For example, while it is necessary to ask specific questions to test diagnostic hypotheses, this can easily lead to confirmation bias. Let’s say a clinician asks patients with frontal headaches if they have nasal congestion, and 40% of them (to use a hypothetical figure just for illustration) say that they do. This might lead the clinician to conclude that this is an important finding, and that it confirms that many patients with frontal headaches probably have sinus headache. What we do not tend to do, however, is institute a control – in this case ask patients with posterior headaches if they have nasal congestion, or patient without headaches. Perhaps 40% of all patients report nasal congestion.
This fallacy is similar to the toupee fallacy – believing that one can always recognize a toupee. This may be due, however, to not being aware of (and therefore not counting) instances when you cannot recognize a toupee. In diagnostic terms, signs and symptoms may seem to correlate strongly with a certain clinical presentation, but that can be an illusion of confirmation bias; only looking for the correlation to confirm your suspicions.
This is exactly why we need controlled data to guide our clinical assessments. What the clinician really needs to know, for example, is what the predictive value of having sinus congestion is for someone who presents with a frontal headache.
Another way to look at this is that anecdotal experience can be misleading, while carefully collected scientific data is reliable and predictive. This not only applies to questions about which treatments are safe and effective, but to the application of those treatments to individual patients.
Clinicians are also prone to correlation fallacies – assuming causation from correlation and temporal sequence (post hoc ergo propter hoc). This may be especially true for apparently good responses to our treatments. If I give that patient a treatment for their headaches and the headache gets better, it is very tempting to take credit for the improvement. We need to recognize, however, that the improvement may have been entirely coincidental, or due to placebo effects. This has practical implications going forward. If the headache recurs, will the same treatment also be effective? If the treatment has side effects, are they worth it? There may also be diagnostic implications – if I gave the patient a migraine-specific treatment and they improved, does that mean they have migraines?
There are other aspects of gathering the patient’s medical history that require the constant application of skepticism. First, the history is being filtered through a person (usually the patient, but perhaps a family member or other caregiver). This means the history is subject to all the flaws of human memory. Memories can be confused, fused, and altered over time. Every time you remember something you are actually reconstructing and potentially changing the memory. This process is exacerbated with a medical history, and a patient may have told their history to many individuals. There is therefore a large potential for contamination. Every time an interviewer asks a question they are potentially contaminating the patient’s memory with the content of that question.
There are also phenomena known as source amnesia and truth amnesia – we are better at remembering facts than we are at remembering where they came from (the source) and whether or not they are even true. So patients may relate a lot of information about this prior history, workup, treatment, and diagnosis while being confused about where the information came from, or even if it is true or not.
Clinicians can also make the mistake of assuming that because a bit of patient history is written down in the chart it is more likely to be true. This is not necessarily the case, however. There is what we call “chart lore” – facts passed on from one clinician to the next as part of the history but never traced to the original source. Each person just assumes the last person checked it out before they wrote it down. I frequently see patients who are walking around with medical diagnoses they don’t have. One clinician told them they might have a diagnosis, or even that the chance that they have it is very low, or perhaps that they don’t have a diagnosis. They, however, leave with the impression that they do have the diagnosis, and they pass that on to the next clinician they see, who writes it down as part of their history.
Further, we need to recognize that each patient has their own narrative – the way in which they understand and perceive their own health and illness. This narrative then affects all the information they have about their illness – what they choose to reveal, what they don’t reveal, the timing of events, the chain of cause and effect, and what labels they accept. The clinician has to simultaneously understand this narrative and investigate what is objectively going on that has led to this narrative. The narrative is important because it will influence how we communicate to the patient and make treatment decisions with them.
But we should not confuse this narrative with reality. The job of the clinician is to be detached, to step back and try to look through the narrative, the chart lore, the logical fallacies, imperfect memory, and all the rest to figure out what is really going on. In other words, the clinician has to be skeptical – skeptical of every piece of information and how it is interpreted, and skeptical of themselves and their own biases and desires.
This, of course, is an idealized goal, something we need to continuously strive for but probably never attain. Perhaps the most pernicious aspect of the current infatuation with unconventional treatments is that they overtly promote gullibility rather than skepticism on the part of the clinician (and everyone else). This is partly why unscientific medicine cannot be “integrated” into science-based medicine – they are fundamentally philosophically incompatible. Practicing science-based medicine requires rigor and skepticism, while the “alternative” requires a surrender to gullibility and naivete.

Pseudoscience is not Cost Effective
Steven Novella on May 9, 2012
Link
Industrialized nations are in the middle of a health care crisis (some more than others), or at least a dilemma. As our medical technology advances, people are living longer, they are living with chronic diseases, and they are consuming more health care. The cost of this health care is rising faster than economic growth, so it is becoming a greater and greater burden on society. Many countries ration health care in one way or another in order to contain costs. Otherwise there is no easy or obvious solution and it’s likely that difficult choices will have to be made.
An interesting side effect of this dilemma is a renewed focus on the cost effectiveness of medicine. Effectiveness alone is not enough. We simply cannot afford, for example, to introduce a very expensive treatment for marginal improvement in outcome in a common disease. Different options can also be compared not only for their safety and efficacy, but for their cost effectiveness. In other words, we need to use cheaper alternatives when available rather than always reaching for the latest and greatest (and most expensive) treatment.
This situation provides an opportunity for science-based medicine. Treatments that are promoted as complementary and alternative (CAM) are often sold as cost effective because they are less expensive up front than standard medical care. We cannot, however, cede this argument to proponents of dubious therapies. Cheap does not mean cost effective. You have to be effective in order to be cost effective, and most of the dubious treatments that are marketed under the CAM umbrella are ineffective.
Regulators in Australia seem to get this. ABC news (that’s the Australian Broadcasting Corporation) recently reported:
Natural therapies found to be clinically ineffective will be cut out of government-funded private health insurance rebates.
Treatments including homeopathy, aromatherapy, ear candling, crystal therapy, flower essences, iridology, kinesiology and naturopathy could be found ineligible.
This is good news, however I think labeling this as focusing on “natural therapies” is counterproductive. Such labels are often misleading, inaccurate, meaningless, and a distraction from what really matters – the scientific evidence. What is “natural” about sticking a candle in your ear and burning it to suck out the toxins (which look suspiciously like burned candle wax)? “Natural medicine” is just a marketing term without a useful or meaningful operational definition. Proponets of these dubious methods are using the label to criticize this measure. Also from the ABC article:
However, Australian Traditional Medicine Society president Dr Sandi Rogers says the announcement came as a surprise.
“It’s a little bit of a shock when we as a profession have not been consulted,” she said.
“If this cost-cut is saying ‘we don’t want to spend taxpayer’s money on natural medicine’, I would be very concerned.”
They are spinning this as an attack on “natural medicine.” Rather, the measure is saying that taxpayer money should not be spent on therapies that are not adequately science-based, whether or not they are thought of as “natural”, “traditional”, or “conventional”. All of the modalities listed above are highly implausible and without evidence to support their efficacy.
Concerns about cost effectiveness and public funding are a great opportunity, in fact, to have a public discussion about the efficacy of such treatments. I want everyone to know exactly what homeopathy is (implausible treatments based on magical thinking diluted into non-existence), and I want them to know what the scientific evidence says – that it doesn’t work. Let us then have a frank debate about whether or not the FDA should be approving homeopathic potions, and whether our limited public health care dollars should be wasted on them.
This comes back to the notion that there should not be any double standard when it comes to medicine. CAM proponents usually try to turn this around, claiming they are not being treated fairly. Dr. Rogers, for example, is quoted as saying:
“We would just like a fair playing field.”
I don’t believe that is true. CAM proponents want a double standard with unfair advantages given to so-called CAM therapies. That is the real purpose of the existence of such labels, all created by proponents in order to argue for the double standard. Defenders of science-based medicine are arguing for a single science-based standard in evaluating medicine. We should apply this same standard when considering cost-effectiveness and public funding. CAM therapies should be held to the same standard of plausibility and scientific evidence, and not be given special consideration because they are “natural.” CAM proponents should also not be allowed to change the rules of evidence as they go along in order to rig the game in their favor.
What is amazing is that public rebates were being given for things like iridology (a completely pseudoscientific form of diagnosis), not that they are now going to be taken away.
The cost effectiveness debate is a good opening for proponents of SBM to make the case to regulators that they should not be wasting taxpayer health care dollars on treatments that are not supported by evidence. I hope this the start of a trend. The principle is very simple – we cannot continue to waste resources on pseudoscience in medicine.

Who you gonna believe, me or you own eyes?
Mark Crislip on July 12, 2013
Link
Mrs. Teasdale: Your Excellency, I thought you’d left!
Chicolini: Oh no, I no leave.
Mrs. Teasdale: But I saw you with my own eyes!
Chicolini: Well, who you gonna believe, me or your own eyes?
Duck Soup. Funniest movie ever.
If I could choose a super power, it would be neither flight nor invisibility, but the ability, like Triad, to separate into multiple people so I could accomplish more. I find that my multiple personality disorder is not all that efficient at getting things done. The Goth cowgirl? Lazy.
So sometimes I have to cut corners. As this post goes live I am at TAM helping with panel discussions and workshops and the only way I can get a post up is to cannibalize my lecture. Dr. Gorski will not let me post the slides and be done with it; those managing editors can be so unreasonable. Full sentences. Proper spelling. Good grammar. Sheesh. Some people.
The topic of my presentation is the cognitive errors that lead people to believe in nonsense and is, or was, a brief tour of the flawed ways in which we think and how the brain allows everyone to be under the false impression that fictions are real.
In the old days I simplistically thought people were just stupid, uninformed, or both. With 45% of Americans believing in faith healing, 37% in astrology, 30% in UFOs and 25% in reincarnation (if true I am sure the fates will bring me back as a rabbit in a syphilis lab), it was just that people are ignorant dumb-asses, right? Give people the facts and they will realize that they are wrong an alter their opinion accordingly. Right? Nope.
I have met the ignorant dumb-ass and he was me.
Pogo. Sort of.
Most folks are neither stupid nor dumb-asses. Critical thinking is not default mode of the brain and most of the time, for most people, critical thinking is a waste of time. It is for me. For the activities of daily living there is little need to think critically. We rely on our experience and the experience of others to decide what to do. It is often an invaluable short cut. I want to eat out, I check out Yelp. I want new music, I ask my kids (except the hip-hop. Tats, hip-hop and square glasses are some of the styles that confirms I am old. I don’t understand the aesthetic of any of them). I read many of the reviews on Amazon before ordering a product. And I never bought a car because it was highly rated on consumer reports. I get the car that elicits a frisson of want and I have enjoyed every car I have owned.
The first principle is that you must not fool yourself—and you are the easiest person to fool. So you have to be very careful about that. After you’ve not fooled yourself, it’s easy not to fool other scientists. You just have to be honest in a conventional way after that.
— Richard Feynman
It is fun to quote Feynman, but for day to day life it rarely applies.
And then you get to health care. Life and death, sickness and health. And for probably the first time the paradigm by which everyone interprets the world, experience and the advice of others, is no longer applicable or reliable for patient or doctor. It is no wonder that people trust the anecdotes and narratives of SCAM users and providers. They are using the methods most of us use to evaluate the world.
I like to say the three most dangerous words in medicine are ‘I lack insurance’. No. That’s not it. It is the words ‘In my experience.’ But experience dominates over critical thinking every day in every way. If I liked to participate in the naturalistic fallacy, I would say critical thinking is un-natural, like plastic, cement and Twinkies (I suspect all are made in the same factory from the same material), a man-made construct never found in the wild.
Feynman was a genius; most of us are not. W was closer to the human condition:
There’s an old saying in Tennessee — I know it’s in Texas, probably in Tennessee — that says, fool me once, shame on — shame on you. Fool me — you can’t get fooled again.
The lists on Wikipedia of cognitive biases, logical fallacies, and memory biases are sobering. There are so many ways to think poorly it is remarkable we get anything done. Everyone has their favorite fallacies. I like:
    • Focusing effect: the tendency to place too much importance on one aspect of an event; causes error in accurately predicting the utility of a future outcome.
    • Confirmation bias: the tendency to search for or interpret information in a way that confirms one’s preconceptions.
    • Illusory correlation: inaccurately perceiving a relationship between two events, either because of prejudice or selective processing of information.
    • Clustering illusion: the tendency to see patterns where actually none exist.
These are arguably the most important fallacies that allow people to see efficacy in nonsense. SCAMs would have difficulty existing without them.
I am terrible at recognizing logical fallacies in realtime. (I always fail when they do Name That Logical Fallacy on SGU. Or is that conformation bias?) When I do note fallacies in others, I discover that people do not take my observation that they are thinking poorly with grace and gratitude.
Go figure. People do not appreciate having their intellectual flaws identified. It is like telling them they have no sense of humor or are a lousy driver. It is one of the reasons rational opinions are ignored. No one likes a know-it-all and we all are aware of what ultimately happened to Mr. Know-It-All.
Not only do people not like having their intellectual shortcomings noticed, they probably are unable to recognize the fact that they are not excellent thinkers. So much of life is explained by the Dunning-Kruger effect:
The Dunning–Kruger effect is a cognitive bias in which unskilled people make poor decisions and reach erroneous conclusions, but their incompetence denies them the metacognitive ability to recognize their mistakes.
The unskilled therefore suffer from illusory superiority, rating their ability as above average, much higher than it actually is, while the highly skilled underrate their own abilities, suffering from illusory inferiority.
Actual competence may weaken self-confidence, as competent individuals may falsely assume that others have an equivalent understanding. As Kruger and Dunning conclude, “the miscalibration of the incompetent stems from an error about the self, whereas the miscalibration of the highly competent stems from an error about others.
The effect is about paradoxical defects in cognitive ability, both in oneself and as one compares oneself to others.
Ever see a second year surgical resident treat a bacteremic Staphylococcus aureus infection with clindamycin? Orally? And underdosed at that. I have. Many times. They have a culture and susceptibility, what more is needed to treat an infection? They have no clue that they do not know a burro from a burrow when it comes to treating infections. Kind of scary.
Combine Dunning-Kruger with the Peter Principle and history is explained far better than Das Kapital or The Foundation Trilogy.
And then there is memory. It is remarkable how flawed our memories are. In my ignorant youth I had though the memory of my life was like Super 8 film, or, perhaps for you youngsters, a youtube video: a perfect recording of events. One of my intellectual epiphanies was the book The Seven Sins of Memory by Daniel Schacter. I had no idea just how unreliable my memory is and how much of my past is a constructed narrative.
The Sins
Sin 1: Memory fades
After a month 75% of the memory of an event fades. Except the lyrics of songs from high school. Those ARE forever. Most recollections of past events are reconstructions based on current expectations and knowledge: people remember the past not how it was but how they think it should have been.
Sin 2: Misattribution
We remember events that never happened or attribute events to people and things that were not there, or recall what happened but it occurs at the wrong time and place. One of the many reasons “anecdotal evidence” of therapeutic efficacy is suspect.
Sin 3: Memory is suggestible
More than one-third of subjects recalled being hugged by Bugs Bunny at Disneyland. Impossible because Bugs is not a Disney character – after a researcher planted the false memory.
If you don’t accurately remember whether an SCAM worked, and you think it should, and someone tells you the SCAM worked, you will remember that it was indeed effective.
Sin 4: Memory is biased
I call it the Gigi effect. The whole thank heaven for little girls thing is kind of creepy now, but we have all had, shall we call it a discussion, with our significant after a social event and we both remember the events under disagreement in ways that make us look noble and the other suspect. And if you think a SCAM is the next best thing to champagne, then you will remember the intervention fondly.
Sin 5: Memory has persistence
Especially when associated with stressors. Medical training has left me with TNTC memories associated with a wee bit of PTSD. I still have not-ready-for-the-test dreams. Intense experiences imprint memory and give them disproportionate importance later. I can be far more biased by failures and complications than any good outcome.
Health issues are major stressors, so the flawed memories are going to have a disproportionate impact.
To be complete, 6 and 7 are absentmindedness and blocking, but are not germane to this discussion, but I know if I did not include them there would be HE-doublehockeysticks to pay. Our readers are a fastidious lot.
It all comes together in the archetype, at least for research, in N-Rays. I love the story of N-rays because it is in the hardest of the hard sciences, physics. Blondot was a French physicist who thought he saw changes in the brightness of an electric spark that he thought as due to a new form of radiation, naming it the N-ray. 120 other scientists had over 300 published articles claiming to be able to detect N-rays emanating from most substances.
Most researchers at the time used the perceived light of a dim phosphorescent surface as “detectors”, although work in the period clearly showed the change in brightness to be a physiological phenomenon rather than some actual change in the level of illumination (Wikipedia).
It was suspected at the time that Blondot was seeing things that were not actually there since the observations made no sense given the understanding of the times. A killjoy physicist, Robert Wood, visited the lab, surreptitiously sabotaged the N-ray machine, yet Blondot et. al. still saw the N-rays.
The modern equivalent is people who have adverse effects from cell towers even when the towers are off or who have salubrious effects from magnets even when there is no magnetism. The ability to experience what is not there is astounding.
The last thing that I have learned is that for many people the facts just do not matter. People will hang on to their beliefs no matter what the evidence: derp.
English has no word for “the constant, repetitive reiteration of strong priors”. Yet it is a well-known phenomenon in the world of punditry, debate, and public affairs. On Twitter, we call it “derp”.
Which is to say, a policy commentator is “derpy” when his or her (usually his) prior assumptions about the world are so unwarrantedly strong that he is unswayable by evidence. Derpers have a faith-based approach to policy.
When I asked my youngest son about derp, he groaned and said something to the effect that my mentioning the word was the definition of cringe-worthy and to never do it again. Just a warning for those of you with teenagers at home.
My favorite one star review of my Quackcast in iTunes was:
Harm to the brian (sic).
Didn’t need to listen.
No facts for him. Derp.
And Brian? So Sorry.
So when someone mentions errant information in support of a particular SCAM, your response with some reality based fact will likely go nowhere. I suspect the key word is ‘actually’. Start a sentence with ‘actually’ and you might as well stop there. They do not want to hear what you have to say and will not consider it valid even if they hear it. If ‘actually’ was removed from the skeptical lexicon, we would never get a sentence started in a discussion with a woo believer.
Human nature predisposes us to believe SCAM. You can’t change human nature but you can be aware of its flaws and compensate.
It is, actually, what makes a skeptic and a critical thinker
BTW: Some of this post was cut and pasted from old Keynote presentations without references. I do not think I am plagiarizing other authors, but I am not certain if all the words are indeed mine, but Google searching suggests they are.

Shedding Light on Unreasonable Decisions
Harriet Hall on January 20, 2015
Link
One of the biggest frustrations for a doctor is when a patient refuses to take science-based medical advice. We would like to believe that giving a patient accurate information will lead him to make good decisions that will improve his health or save his life. But that’s not how it works. Patients reject life-saving surgery and chemotherapy, patients on essential medications are non-compliant, parents reject vaccines for their children…what are these people thinking? Why would anyone in their right mind knowingly reject a treatment that has been proven to increase their chances for survival and health? What could their reasons possibly be?
This ties into a subject we have debated over and over: why do people choose alternative medicine? Many reasons have been suggested: cost and accessibility, the need for control, dissatisfaction with mainstream medicine, the peer pressure of a popular fad, “belonging” to a group of like-minded people, a need for answers, autonomy, health freedom, ideology, rebellion against authority, a need for hope even if it is false hope, giving more importance to stories than to studies, the post hoc ergo propter hoc fallacy, scientific illiteracy, misinformation, superstition, magical thinking…the list goes on. Studies have been done, but we can’t be sure the reasons people give to researchers are the real reasons. There is a problem with the search for reasons: these decisions are not made on the basis of reason. Physician Lisa Rosenbaum has written a beautiful essay in The New England Journal of Medicine entitled “Beyond Belief — How People Feel about Taking Medications for Heart Disease“, that sheds a penetrating light on what is really going on. It made me think of the subject in a whole new way.
Flavio’s tophi
To put a human face on this issue with an example, let me tell you about a memorable patient I once had (I’ll protect his privacy and call him Flavio). It was the mid-1970s. I was an Air Force doctor stationed at Torrejón AFB just outside Madrid, Spain; Flavio was an Italian who had chosen to spend his golden years in Spain. Somehow (I never learned his story) after a childhood in Italy he had ended up serving 20+ years in the American military and was authorized medical care at our hospital as a military retiree. I didn’t think of it until just now, but he was following in a time-honored two-millennia-long tradition – Roman soldiers used to retire to Spain. In fact, the city of Mérida in western Spain was founded in 25 BC by the Emperor Augustus as a colony for old soldiers discharged from his army; the original name was Emerita Augusta, meaning Augustus’ veterans. Flavio still had a bit of an Italian accent, and he was a delightful, charming, old-school gentleman. It was a pleasure to take care of him; he was always pleasant, courteous, and grateful for every little thing I did for him. Flavio had a bad case of gout, with big lumps on his elbows and elsewhere that were easily recognizable as tophi, deposits of uric acid around the joints that are a pathognomonic sign of gout. I put him on medication; his pain resolved and the lumps went away. Since the lumps were no longer bothering him, he stopped the medication. Predictably, the lumps came back. I put him back on the medication and emphasized that he needed to stay on it. The lumps shrank; he stopped the medication again; the lumps came back; he came back to see me. This sequence of events repeated itself over and over. My explanations and exhortations had no impact whatsoever. He had seen for himself that the medication worked; he had seen the lumps come back when he stopped taking it; he had heard and understood my explanation of how it worked and why he needed to keep taking it; he wasn’t stupid; he wanted to be pain-free and lump-free. But he kept stopping the medication. I simply couldn’t understand. It puzzles me to this day.
The Needlephobe’s story
And then there was my encounter with a man who refused to get a flu shot. I asked him why, and he gave me reason after reason that I quickly shot down with the facts. Finally his excuses were exhausted. He had to agree that there was no reason not to get the flu shot, but he still wasn’t going to get one because he just didn’t like needles. People reason after the fact; they decide and then they look for reasons to support that decision. If they don’t really understand their own decision process, how can they hope to explain it to us?
Rosenbaum’s quest
Rosenbaum is a cardiologist. After a heart attack, only about half of patients are compliant with taking medications to prevent a recurrence. Instead of just looking for reasons, she set out to find out how people feel about taking medications for heart disease. She learned that affective responses leave people far more sensitive to possibility than to probability, and that people who have had drug side effects tend to have an increased perception of drug risks and a decreased perception of their benefits. Taking medication reminds people that they are sick; they would rather deny their problems and think of themselves as healthy. They may prefer “natural” remedies to pharmaceuticals on principle. After the wake-up call of a heart attack they adopt a healthier lifestyle and their new sense of well-being may convince them they don’t need pills. After short hospitalizations for heart attacks, sometimes as little as 24 hours, today’s patients are not as impressed that they have a serious problem as earlier patients were after 4-6 weeks of bed rest, sometimes complicated by bed sores, arrhythmias, and depression. Rehabilitation programs create a supportive environment where people can accept their disease and also feel healthy, but only 14-35% of patients participate in those programs. The benefits of medications aren’t obvious: if patients don’t “feel” better, they may believe the medications aren’t doing anything. Framing helps: adherence may be improved if patients think of the medications as “keeping the pipes open.” Fear of chronic illness sometimes trumps fear of death. Some people feel they should be able to completely control their own problems with lifestyle changes; they see medications as a sign of weakness, a cop-out.
Her final paragraph is a gem:
It’s our job to help patients live as long as possible free of complications of cardiovascular disease. Although most patients share that goal, we don’t always see the same pathways to get there. I want to believe that if patients knew what I know, they would take their medicine. What I’ve learned is that if I felt what they feel, I’d understand why they don’t.
Conclusion
Those last words should be engraved in stone. Humans are not like Mr. Spock on Star Trek: they don’t make healthcare decisions based on cold reason. Instead of asking “why?” we would do better to ask “how do you feel?” We would like to believe that if people only knew what we know, they would make the same decisions we would. But if we felt what they feel, we’d understand why they don’t.

How to Talk to People About CAM
Harriet Hall on October 22, 2013
Link
Recently a correspondent asked me for advice about his parents. He said they use things like homeopathy, acupuncture, and copper bracelets. They use conventional medicine too, but it seems to be a 50/50 approach that gives each an equal weighting. He has tried to talk to them about things like homeopathy and the placebo effect, but the shutters come down hard and fast. He tries to criticize the alternative treatment itself without offending or attacking the person, but his mother still sees it as a personal attack. He worries that as they get older and in need of more medical care, his parents may not make the best decisions. He asks about how to tactfully have these conversations and perhaps change their point of view.
That’s a very tough question that gets asked a lot, and I don’t have any good answers; but I do have some thoughts and untested ideas that could serve as the starting point for a discussion, and I hope readers will pipe up in the comments and tell us what has or hasn’t worked for them.
In the first place, you can’t change another person’s mind. They can only change it for themselves. If you can get them thinking to where they have an “Aha” moment and realize something for themselves, that’s far more effective than trying to tell them. I bet Socrates would have been good at that, but I’m certainly not. I did succeed once by accident. I had written an article about acupuncture in Skeptic magazine, and an acupuncturist wrote to me afterwards. He said:
Your article about acupuncture made me angry. I thought that you had to be wrong because (of course) I had heard about so much research about the evidence for acupuncture… got me to actually look at the literature instead of just ‘knowing’ that it exists. In trying to prove you wrong I proved myself wrong. One textbook made a claim that acupoint such and such is good for stimulating the pituitary to produce anti-diuretic hormone for patients with polyuria. I thought, “wow that’s really interesting I wonder where the reference is.” I looked for a reference and was struck by the fact there were no references at all anywhere in the book for any of the claims.
I have gradually let go of the belief that acupuncture has any basis for treating anything.
He decided to change his occupation to something based on reality. He was a rare person to be able to question the basis of his livelihood and change his mind in response to facts. All I did was create an opportunity (by making him angry) for him to think for himself.
You can’t change a person’s mind with science if their beliefs were not based on science in the first place. They may not understand or trust science. We are tempted to think that if someone only knew the facts as we know them, they would have to agree with us; but it doesn’t work that way. Instead of throwing science at them, you might try to understand their thinking. How did they come to form those beliefs? Can you trace the thought processes that led them to use CAM? Is CAM filling some emotional need for them? Does it give them hope? Do they feel frightened and helpless in the face of seemingly unpredictable disease? Do they “need” to believe that they can have some degree of control over their own health? That they can control their own fate rather than submitting to the whims of kismet? Did they have a bad experience with an MD? Are they rebelling against a medical system that betrayed them in some way? Are they frustrated because conventional medicine can’t offer a cure or complete relief of their symptoms? Did they have side effects from pharmaceuticals? Do they like to try new things and experiment? Do they hate to be told what to do? Do they have a knee-jerk rebellion to any source of authority? What have they been told about CAM? Were they influenced to try CAM by someone they trust implicitly? Do they have some basic misunderstandings of anatomy or physiology? Do they have any understanding of the many ways our minds tend to lead us to falsely conclude that a bogus treatment works? Unless you can get inside their head and understand how they think, what experiences they have had, what they don’t know, and what they know that isn’t so, it’s hard to communicate on a common level.
Do they have good critical thinking skills about other subjects? It might be useful to ask if there any kind of quackery they reject. If so, you could ask them to explain why they rejected that one but don’t reject others. Do they make fun of any of the late-night TV infomercial scams or laugh at advertisements for miracle products that make promises too good to believe? Do they distrust used car salesmen? Do they know anyone who has been victimized by a con artist? Are they skeptical about anything like Bigfoot, psychics, 9/11 conspiracy theories, AIDS denialism, anti-vaccine propaganda? Would they accept a subscription to either of the magazines Skeptic or Skeptical Inquirer to read about the things they are skeptical or curious about? That might incidentally expose them to articles about things they are not currently skeptical about. Do they accept the advice of Consumer Reports for buying cars and appliances? Would they read Consumer Reports on Health?
I had an illuminating experience involving dowsing. I met a man who firmly believed in it, and I lent him the book Water Witching USA, by Evon Vogt and Ray Hyman, which explains all about the ideomotor effect, the forces acting on a forked stick, proof that the dowser is moving the stick without realizing it, the tests that have repeatedly and consistently shown that dowsing doesn’t work, the psychology of belief, etc. It’s a skeptical classic. I naively expected him to give up his belief when he read the same facts that had made me reject dowsing. We were scheduled to give a public presentation with him on the Pro side and me on the Con side. I was dumbfounded when I realized his presentation boiled down to only two arguments: he had seen it work, and lots of other people believed in it. I presented a summary of the information in the book, but no one was impressed. One woman commented afterwards, “I wanted to hear more about how it works and less about how it doesn’t work.”
I know better now. Facts and rational arguments have no effect on a true believer. People hear only what they want to hear. Most people refuse to listen to anything that contradicts their beliefs. Their confirmation bias accepts only confirming evidence to protect their ego from having to admit they were wrong. And there’s a real danger in arguing with them. They will come up with rationalizations that allow them to reject everything you say, and that only serves to reinforce their beliefs. They come out of an argument believing even more strongly than when they went in.
You could tell them that you are willing to change your mind, and explain what it would take to make you change your mind and accept that a CAM treatment was safe and effective (Essentially the standards for scientific proof: positive scientific studies that are well-designed and replicated, leading to a scientific consensus of experts based on a critical evaluation of all the published evidence; with more rigorous standards of proof for treatments that don’t have a plausible mechanism of action). Then you could ask them what it would take to make them give up their belief: what kind of evidence would convince them? What if several well-designed new studies found strong evidence that it didn’t work? I’ve had people say that couldn’t possibly happen, because it works. End of discussion.
If you read about someone who was harmed by X, it wouldn’t hurt to share the information as long as you preface it with something like “I read this and it made me worry about you. I’m not asking you to stop using X, but I know you care about me and how I feel, and I wanted you to understand why I feel this way and why I’m so concerned.”
Drops of water can eventually wear away stone, and they may also hear criticism of CAM from other sources and eventually begin to doubt some of their beliefs. You may be planting a seed that others will water and eventually a sprig of doubt may sprout.
There is hope. On both Quackwatch’s Healthfraud discussion list and in the SBM comments, quite a few people have said that a constant exposure to fact and logic on those websites had gradually had an impact over time. They said that they had learned how to think more critically about CAM and had given up prior beliefs.
Some people are true believers who will never change their minds. There’s really nothing we can do to alter their religious-like beliefs, just as you couldn’t expect to argue a Catholic or a Muslim out of their beliefs and into atheism. But lots of people are fence-sitters; they may not really have thought seriously about the subject before, or they may have heard arguments on both sides and still be undecided. If they haven’t irrevocably made up their minds, they may listen to what we have to say.
There’s a flip side to this question. If you have cancer, how do you answer well-meaning friends who want you to stop your chemotherapy and try their brand of snake oil instead? You can’t very well say you think their advice is stupid. How can you reject their advice without damaging the friendship? Is there some diplomatic sound-bite answer we could keep in mind for those occasions?
When talking with a loved one, it’s important to emphasize that you love the person and only hate what they are doing, and it’s important to help them understand why you are worried about them. Let them know that you will have to agree to disagree, but that you respect their right to make their own decisions, and that you will continue to love them and support them no matter what they decide or how unhappy it makes you.
Conclusion
As I said, I don’t have answers, just thoughts and suggestions. Maybe you can help. Let the discussion begin!

Making One’s Own Reality – Food Babe Edition
Steven Novella on March 18, 2015
Link


The default mode of human activity is to construct our own internal model of reality based upon our desires, biases, flawed perceptions, memories, and reasoning, and received narratives from the culture in which we live. That model of reality is then reinforced by confirmation bias and jealously defended.
But we also have the capacity to transcend this pathway of least resistance. Philosophy is the discipline of thinking carefully and systematically about ideas to see if they at least are internally consistent. Science is the discipline of systematically and carefully comparing our internal models of reality against objective reality, and then changing those models to suit the evidence.
Everyone engages in a combination of bias, superstition, logic, and evidence-based reason to varying degrees – the question is, to what degree? The goal of science-based medicine is to increase the proportion of science and reason in the mix with respect to the practice of medicine and public health.
There are many forces at work in society, however, that explicitly oppose the role of science because, in my opinion, they find it inconvenient to their internal model of reality or whatever narrative they are selling.
At its core, we are engaged in a cultural battle over reality. We advocate a transparent, fair, and rigorous science-based approach to determining what is reality. Others, however, are trying to break or replace the rules of science so that they can advocate for a version of reality that is consistent with their narrative.
This has probably always been part of the human condition, but seems to be more acute recently, likely because of the power of the internet and social media. It is quite easy these days to create an alternative infrastructure dedicated to promoting a particular version of reality.
If you are, for example, an anti-vaccine ideologue, you can have your own experts, studies, journals, websites, and networks, creating an echochamber dedicated to your version of reality.
Sometimes these echochambers arise spontaneously as part of a grass-roots movement. Sometimes they are manufactured deliberately as part of a campaign to erect a pocket of alternative reality in our society, or to defend a vested interest.
We address many examples of manufactured alternative realities on SBM, and I am going to focus on one today for the purpose of illustration – Vani Hari (aka the Food Babe).
Hari is an up-and-coming food activist (although she is getting to be old news rather quickly). We have discussed her here and on other science-based blogs because she appears to be an utterly scientifically illiterate ideologue.
Her narrative is that all chemicals are bad, anything synthetic is bad, and anything that superficially seems yucky to her must also be bad. She has literally stated that there should be no amount of any chemicals in food. She has also famously stated that any substance whose name she cannot pronounce should not be in food.
She is, unfortunately, media savvy and has been very effective at self-promotion. Her routine is to identify something she finds yucky in food, completely misrepresent the science, then fearmonger about the safe ingredient in order to bully companies into doing PR damage control. Recently, however, the scientific community is noticing how dangerously scientifically illiterate she is.
Last week The New York Times published an unflattering article about Hari. Here is a typical passage:
Kevin M. Folta, the chairman of the horticultural sciences department at the University of Florida, described Ms. Hari’s lecture at the university last October as a “corrupt message of bogus science and abject food terrorism.” (Her fee was $6,000.) Dr. Folta added, “She found that a popular social media site was more powerful than science itself, more powerful than reason, more powerful than actually knowing what you’re talking about.”
The article also quoted our own David Gorski:
David H. Gorski, a surgical oncologist who also has a degree in chemistry, wrote on Science-Based Medicine that the beer ingredient is propylene glycol alginate, which, despite its name, is not even close to propylene glycol, is not antifreeze and is derived from kelp.)
Hari recently responded to the NYT article, which is more interesting than the article itself because it demonstrates, in my opinion, her utter intellectual dishonesty – something which is necessary to maintain a bubble of alternative reality.
In her response she blatantly plays the shill card. She tries to dismiss her critics:
as manufactured by “the processed food lobby” and “industry-funded science.”
In her reply she states about Kevin Folta:
Ms. Rubin quotes Kevin Folta, chair of the Horticultural Sciences Department at the University of Florida. He does not specialize in health or nutrition; rather he is a crop scientist specializing in GMOs who seeks industry funding and support for this research (3).
This, of course, answers none of Kevin’s criticisms of Hari. He is a publicly funded geneticist without any conflicts of interest. He would like to see some of the genetically modified plants he has studied make it to market (like disease resistant strawberries) – how does that in any way affect his criticisms of Hari’s unscientific statements?
She also goes after David, stating:
The piece mentions Dr. David Gorski, who also has received pharmaceutical industry funding (6). He likes to vilify “quackery”, and has even attacked the venerable Cleveland Clinic for using some alternative medicine (7). He is simply wrong that there is no propylene glycol, the antifreeze kind, in some beers and alcohol. We know Fireball Whiskey contains it (8) and it’s listed as an approved ingredient in alcohol on the government website ttb.gov (9). This is easy to prove if you aren’t looking to discredit someone. I answered this previously in a response to my critics back in December 2014 (10), but this was ignored and has been re-hashed time and time again.
These statements are, how we say, “factually challenged.” David has responded on his own blog, but to reiterate a couple of points here: David received a small amount of pharmaceutical company funding years ago. This hardly amounts to a conflict of interest in pointing out Hari’s bad science. This is not even part of the food industry. It is easy to see how Hari uses any sliver of an appearance of a conflict as a witch hunt to discredit her critics.
Hari also tries to pull one over on her readers with the propylene glycol issue. She clearly mistook propylene glycol alginate in beer for propylene glycol, because she doesn’t understand that these are different chemicals. The fact that there is a small amount of propylene glycol in Fireball Whisky is completely irrelevant.
Conclusion
The culture war continues. The Food Babe and others are attempting to distort the process of science in order to promote their alternative world view. Part of the strategy is to dismiss and discredit scientists who know enough to level devastating criticism at their particular brand of nonsense.
We are seeing this from antivaxxers, anti-GMO activists, and others as well. Unfortunately, this controversy can have a chilling effect and make it difficult or unappealing for scientists to engage with the public.
However, we need to engage all the more because of popular nonsense like that peddled by self-proclaimed but entirely incompetent “experts.”

“Science.” You keep using that word. I do not think it means what you think it means.
David Gorski on May 11, 2015
Link

I’ve discussed on many occasions over the years how antivaccine activists really, really don’t want to be known as “antivaccine.” However, if there’s one thing that rivals how much antivaccinationists detest being called “antivaccine,” it’s how much they detest being called antiscience. To try to deny that they are antiscience, they will frequently invoke ridiculous analogies such as claiming that being for better car safety does not make one “anti-car” and the like. It is here that the Dunning-Kruger effect comes to the fore, wherein antivaccine activists think that they understand as much or more than actual scientists because of their education and self-taught Google University courses on vaccines, that their pronouncements on vaccines should be taken seriously. If there are two antivaccine blogs that epitomize the Dunning-Kruger effect, they are Age of Autism and, of course, the most hilariously inappropriately named The Thinking Moms’ Revolution (TMR). It is the latter of these two that late last week produced a tour de force of Dunning-Kruger, coming, as the most hilariously off-base posts on TMR usually do, from the “Thinking Mom” known as “The Professor.” I shouldn’t be surprised, given her history, but nonetheless it’s worth taking a look at her latest post, Anti-science: “You Keep Using That Word. I Do Not Think It Means What You Think It Means.”
Actually, it does. And if The Professor is going to spend nearly 7,000 words riffing on a bit of dialogue from The Princess Bride, surpassing in verbiage all but a small minority of my posts, it almost makes me want to make this post 8,000 words.
Fortunately, for you, I resisted that temptation and instead merely retort: “Science. You keep using that word. I do not think it means what you think it means.” Then I demonstrate why.
Not surprisingly, “The Professor” feels compelled to begin by asserting her alleged science bona fides. Describing herself as a “geeky physics major” who nonetheless has the temerity to “question” vaccine science, she declares herself “irritated enough by this journalistic trend to rebut to the popular conception of those who question vaccine science as ‘anti-science.'” What appears to have particularly irritated her and sparked this screed is a rather good article by Joel Achenbach from the March issue of National Geographic entitled Why Do Many Reasonable People Doubt Science? “The Professor” is particularly incensed by a passage in the article in which Achenbach makes the case that people who “doubt science” are, as she puts it, “driven by emotion.” Of course, that’s not exactly the argument that Achenbach does make. His argument, as you will see if you read his article, is considerably more nuanced than that. Rather, Achenbach points out observations that have been discussed here time and time again, such as how the scientific method sometimes leads to findings that are “less than self-evident, often mind-blowing, and sometimes hard to swallow,” citing Galileo and Charles Darwin as two prominent examples of this phenomenon, as well as modern resistance to climate science that concludes that human beings are significantly changing the climate through our production of CO2. Not surprisingly, Darwin’s theory is still doubted by many today based not on evidence but rather on its conflict with deeply held fundamentalist religious beliefs.
Achenbach also makes this point:
Even when we intellectually accept these precepts of science, we subconsciously cling to our intuitions—what researchers call our naive beliefs. A recent study by Andrew Shtulman of Occidental College showed that even students with an advanced science education had a hitch in their mental gait when asked to affirm or deny that humans are descended from sea animals or that Earth goes around the sun. Both truths are counterintuitive. The students, even those who correctly marked “true,” were slower to answer those questions than questions about whether humans are descended from tree-dwelling creatures (also true but easier to grasp) or whether the moon goes around the Earth (also true but intuitive). Shtulman’s research indicates that as we become scientifically literate, we repress our naive beliefs but never eliminate them entirely. They lurk in our brains, chirping at us as we try to make sense of the world.
Most of us do that by relying on personal experience and anecdotes, on stories rather than statistics. We might get a prostate-specific antigen test, even though it’s no longer generally recommended, because it caught a close friend’s cancer—and we pay less attention to statistical evidence, painstakingly compiled through multiple studies, showing that the test rarely saves lives but triggers many unnecessary surgeries. Or we hear about a cluster of cancer cases in a town with a hazardous waste dump, and we assume pollution caused the cancers. Yet just because two things happened together doesn’t mean one caused the other, and just because events are clustered doesn’t mean they’re not still random.
We have trouble digesting randomness; our brains crave pattern and meaning. Science warns us, however, that we can deceive ourselves. To be confident there’s a causal connection between the dump and the cancers, you need statistical analysis showing that there are many more cancers than would be expected randomly, evidence that the victims were exposed to chemicals from the dump, and evidence that the chemicals really can cause cancer.
This, of course, is an excellent description of antivaccinationists, except that they no longer accept the precepts of science with respect to vaccines but cling to their “naive beliefs.” They rely on personal experience and anecdotes rather than statistics with respect to the question of whether vaccines cause autism, and no amount of science, seemingly, can persuade them otherwise. The Professor herself is a perfect example of this. She believes herself to be “pro-science” and so she is when science tells her what she wants to believe. When it does not, as in the case of vaccines and autism, she rejects it, spreading her disdain from just vaccine science to all of science. Indeed, her entire post is in general a long diatribe, consisting mainly of three key arguments: the “science was wrong before” trope; the “peer review is shit” trope; and the “pharma shill” gambit.
Achenbach notes that even for scientists the scientific method is a “hard discipline.” And so it is, because, after all, scientists are no less human than any other human being. The only difference between us and the rest of humanity is that we are trained and have made a conscious effort to understand the issues discussed above. We know how easy it is to confuse correlation with causation, to exhibit confirmation bias wherein we tend to remember things that support our world view and forget things that do not, and to let wishful thinking bias us. Even knowing all of that, not infrequently we fall prey to the same errors in thinking that any other human being does. Nowhere is this more true than when we wander outside of our own field, as the inaptly named Professor does when she leaves the world of physics and discusses vaccines or when, for example, a climate scientist discusses vaccines.
The hilarity begins when The Professor cites philosopher Thomas Kuhn’s The Structure of Scientific Revolutions. Even more hilariously, The Professor quotes extensively from the Wikipedia entry on Kuhn’s book, rather than from Kuhn himself. Kuhn’s main idea was that science doesn’t progress by the gradual accretion of knowledge but tends to be episodic in nature. According to Kuhn, observations challenging the existing “paradigm” in a field gradually accumulate until the paradigm itself can no longer stand, at which point a new paradigm is formed that completely replaces the old. Kuhn’s view of science is a fascinating topic in and of itself and I haven’t read that book in many years. However, most scientists tend to dismiss many of Kuhn’s views for several reasons, in particular because Kuhn tends to vastly exaggerate the concept of “paradigm shift.” Particularly galling is his concept of “normal science,” where in the interregnum between scientific revolutions he portrays scientists doing “normal science” (science that is not paradigm-changing) as essentially dotting the i’s and crossing the t’s of the previous revolution. (It’s a very dismissive attitude toward what the vast majority of scientists do.) Indeed, Kuhn’s characterization of the history of science has been referred to as a caricature, and I tend to agree. Certainly, at the very least he exaggerates how completely new paradigms place the old, when in reality when new theories supplant old theories the new must completely encompass the old and explain everything the old did plus the new observations that the old theory cannot. As Cormac O’Rafferty puts it, “The new can only replace the old if it explains all the old did, plus a whole lot more (because as new evidence is uncovered, old evidence also remains).” The best example for this idea that I like to cite is Albert Einstein’s theory of relativity, which did not replace Newtonian physics, but rather expanded on Newtonian physics, which is what relativity simplifies down to when applied to velocities that are such a small fraction of the speed of light that relativistic contributions drop out of the equations because they are so close to zero that it is reasonable to approximate them as zero.
None of these nuances are for The Professor. She misuses and abuses Kuhn to construct a “science was wrong before” argument that is truly risible:
It would seem likely that a journalist writing a high-profile article on science for National Geographic would not only be aware of Kuhn’s work, but would also understand it well. Achenbach seems to understand the evolution of science as inherently provisional and subject to change when new information comes in, but then undercuts that understanding with the claim, “The media would also have you believe that science is full of shocking discoveries made by lone geniuses. Not so. The (boring) truth is that it usually advances incrementally, through the steady accretion of data and insights gathered by many people over many years.”
This statement is patently false. First off, the mainstream media tends to downplay, if not completely ignore, any contributions of “lone geniuses” to science, as exemplified by the 2014 Time magazine cover story proclaiming “Eat Butter! Scientists labeled fat the enemy. Why they were wrong.” Suddenly, everyone was reporting that consumption of fat, in general, and saturated fat, in particular, is not the cause of high serum cholesterol levels and is not in fact bad for you. “Lone geniuses” (also known as “quacks” in the parlance of the old paradigm) understood and accepted these facts 25-30 years ago and have been operating under a completely different paradigm ever since, but it wasn’t until 2014 that a tipping point occurred in mainstream medical circles and the mainstream media finally took note.
No, no, no. This change, which arguably The Professor vastly overstates, came about through the very accretion of knowledge. Moreover, as much as I castigate David Katz for his nonsense on other issues (such as his embrace of homeopathy “for the good of the patient”), he did get it (mostly) right when he criticized this ridiculous TIME magazine article for many shortcomings and exaggerations, not the least of which is that there never was a “war on dietary fat” and the seeming attitude that it’s OK to eat all the fat you want now. In any case, this is not the “paradigm shift” that The Professor seems to think it is. Rather it was a correction, which is what science does. The process is often messy, but science does correct itself with time.
Next up is an attack on peer review, something without which no antivaccine article is complete. Of course, criticism of the peer review process is something many scientists engage in. As I like to paraphrase Winston Churchill quoting a saying about democracy, “It has been said that peer review is the worst way to decide which science is published and funded except all the others that have been tried.” Yes, the peer review process is flawed. However, as is the case with attacks on the very concept of a scientific consensus, when you see general attacks on the concept of peer review, it’s usually a pretty good indication that you’re dealing with a crank. Ditto attacks on the idea of a scientific consensus and glorification of the “lone genius.” There is little doubt that The Professor is a crank. Naturally, she can’t resist including a section that is nothing more than a big pharma shill argument claiming that the science showing that vaccines are safe and effective must be doubted because everyone’s in the pocket of big pharma. No antivaccine article is complete without a variant of that tired old trope.
The Professor ends by arguing that all that tired, boring, old “normal science” (to borrow Kuhn’s term) is wrong and that True Scientists who perform “Revolutionary Science” (quoting Martha Herbert in the introduction to Robert F. Kennedy, Jr.‘s pseudoscientific new anti-thimerosal screed, of all things!) all have “intuition”:
The ability to “utilize this subtlety and context to make important distinctions” that Herbert describes constitutes the difference between the scientific revolutionaries and those who will continue defending an error until long past the point that it has been well and truly proven to be an error. It is an ability that Albert Einstein possessed to a larger degree than most. Einstein felt that “The true sign of intelligence is not knowledge but imagination.” And that “All great achievements of science must start from intuitive knowledge. I believe in intuition and inspiration . . . . At times I feel certain I am right while not knowing the reason.” Interestingly, another well-known scientist whom many consider to have been “revolutionary” was known to place a great deal of emphasis on intuition. Jonas Salk, the creator of the first inactivated polio vaccine to be licensed, even wrote a book called Anatomy of Reality: Merging Intuition and Reason.
The problem that The Professor overlooks is that in science intuition is nothing if it doesn’t lead to results that are supported by data. Moreover, I would argue that what we have here in those who fetishize “intuition” in science is a massive case of that most human of failures of reason: Confirmation bias. We as scientists remember the times when our intuition ended up being validated and forget the almost certainly many more numerous times when our intuition either led nowhere or even led us astray. Yes, even Albert Einstein and Jonas Salk. Indeed, when called out by a commenter for most likely exhibiting confirmation bias about intuition in science and having it pointed out to her that intuition must never trump data, The Professor proves me right:
Confirmation bias on my part, huh? I have to say that that is absolutely untrue. It took me quite a long time to honor and rely on my intuition, and despite the fact that it is proven right over and over again, I STILL sometimes let my conscious mind override it to my regret. I didn’t understand intuition at all when I was younger and thought “hunches” were silly. In all that time, I have virtually never heard anyone say that “I really regret following my intuition on that, while I have frequently heard the latter — often from parents who have lost their children.
No one is saying “I don’t care what the data says.” I have NEVER said that. I read the data. I analyze the data. And I know it DOESN’T say what the mainstream media says it says — ever noticed that you can’t watch a show on TV these days without a few ads for drugs? I’ve read many of those “studies that show vaccines are safe” and I know their limitations — and they are vast. So, nope. Sorry.
So, first The Professor paints herself as a reluctant convert to trusting her intuition. If we’re going to play a war of anecdotes, I could list quite a number of times when I “trusted my intuition” and later regretted it. Her argument here is the very essence of confirmation bias. She remembers the times her intuition led her where she wanted to go, as well as the times she didn’t listen to her intuition and things didn’t turn out well, and forgets the rest. Then, basically, The Professor goes on to rationalize her relying on her “intuition” over data with respect to vaccines and autism by eliminating that cognitive dissonance. She does that by convincing herself that she doesn’t let her “intuition” trump the data in the case of vaccines because she doesn’t think the data show what scientists think the data show, namely that vaccines are safe and effective, do not cause autism, do not harm the immune system, and do not cause all the evils that Dunning-Kruger poster children like The Professor think they do.
Demonstrating her even more inept understanding of epidemiology and medicine, The Professor then goes on to ask:
Anenbach makes the argument that our intuition will lead us astray, encouraging men to get a prostate-specific antigen test, for instance, even though it’s no longer recommended because studies have shown that on a population level the PSA doesn’t increase the overall number of positive outcomes. But there are people whose first indication of prostate cancer was a high PSA result, and those people’s lives may have been saved due to having that test. Who is to say that the person requesting the test will not be among them? In other words, intuition is not necessarily wrong just because it encourages you to do something that is statistically out of the norm or has yet to be “proven” by science.
And who is to say that the person requesting the PSA test won’t be one who is overdiagnosed, who has an indolent cancer that would never have done him any harm during the remainder of his lifespan, but is harmed by unnecessary surgery and/or radiation? That’s the whole point! The Professor picks the good outcomes and ignores the potential for the bad outcome. Confirmation bias! She then twists all this into a massive argument for the precautionary principle, going so far as to cite the discredited Andrew Wakefield and Robert F. Kennedy, Jr., both whose arguments I’ve deconstructed more times than I can remember. (Just enter their names in the search box of this blog and you’ll see.) She also mischaracterizes Achenbach as saying that the scientists most dedicated to truth are the ones who break with the “existing paradigm,” even going so far as to say that by Achenbach’s own words Andrew Wakefield must be more dedicated to the truth than most scientists. Yes, I laughed out loud when I read that part. I also reread his article and could find nowhere where he actually made that argument.
So basically, The Professor ends up arguing that we should be wary of vaccines based on the precautionary principle, ignoring all the many years of accumulated evidence that vaccines are safe and effective and do not cause autism, in favor of the “intuition” of antivaccine activists and the work of “lone geniuses” like Andrew Wakefield because, you know, all that boring epidemiology is “normal science” (to quote Kuhn) and what her antivaccine heros are doing is “revolutionary science.” It doesn’t matter that their “revolutionary science” is wrong, of course, because, you know, The People:
Science can serve corporate interests or it can serve the interests of humanity. There will certainly be places where the two will intersect, but there will always be places where they will be in opposition, and science cannot serve them both. It is most assuredly not “anti-science,” rather it is “pro-humanity” to insist that, where the interests of the two are opposed, science must serve humanity over corporations. We are nowhere near being able to say that is currently the case, however, and while it may be prudent for individual scientists to stick with the tribe in order to further their careers, it is not prudent for us as a human collective to let corporate interests govern what that tribe thinks and does. Until the day that we can say science always puts humanity’s interest first, not only is it prudent for us to question, analyze, and even scrutinize “scientific consensus” from a humanist viewpoint, it is also incumbent upon us to do so.
She even asks if it’s “antiscience” to condemn Josef Mengele’s atrocities and the Tuskegee syphilis experiment in a hilarious rhetorical flourish against a straw man argument that no one—and I mean no one—has ever made. You know, The Professor is starting to sound a lot like a toned down version of Mike Adams. Also, instead of “I’m not antivaccine, I’m pro-vaccine safety,” we now have “I’m not antiscience, I’m pro-humanity.” It’s the perfect toxic combination of Dunning-Kruger married to an outsized ego that thinks that a bit of training in physics trumps the knowledge of people who have spent their lives studying vaccines, autism, and the immune system. In other words, it’s a perfect distillation of the antivaccine movement.

A journey to alternative and integrative medicine apostasy
WIRED posted a story about Jim Laidler recently discussing his movement away from autism biomed and back to science. It’s a good story, that deserves sharing, and explores many of the motivations people have for embracing alternative medicine.
David Gorski on May 4, 2015
Link


I’ve been blogging for over a decade now, a fact that I find really hard to believe looking back on it right now. I’ve told the story before, but it’s worth briefly recounting again because doing so will explain why the story I’m about to discuss caught my attention. My “gateway drug,” if you will, into skepticism was discovering Holocaust denial in the late 1990s on Usenet, a vast and sprawling conglomeration of thousands of discussion forums that began to fade away at the turn of the century with the rise of web-based forums and Google providing an interface to it to make it Google Groups. The forum where I first discovered Holocaust denial and learned to combat it, alt.revisionism, still exists, but long ago degenerated into a cesspool of racism, spam, and trolling. A couple of years later, around 2000 or so, I discovered quackery and the antivaccine movement, thanks to a Usenet newsgroup known as misc.health.alternative, which is where I honed my early skills applying science to medical claims. It’s also where I first encountered Peter Moran, a regular commenter here who greatly inspired me back then with his full-throated criticism of cancer quackery and his website that taught me reasons why cancer quackery could appear to work even when it did nothing to impact the progression of the cancer.
In December 2004, intrigued by all the news stories about blogging and having discovered a number of good blogs, I decided on one dark, gray Saturday afternoon to dip my toe in the blogosphere. On the spur of the moment I created the first iteration of my not-so-super-secret other blog on—what else?—Blogger. Much to my own amazement, over the course of a year I got my little hobby noticed, to the point where I was invited to join a blog collective; by late 2007 I had become prominent enough to be invited by Steve Novella to join this very blog at its founding, where I have remained for seven years, with no plans to move on any time soon. During my early days, though, there was one person who also inspired me, helping me to learn about the pseudoscience that undergirds the antivaccine movement and, in particular, the quackery making up what is known these days as the “autism biomed” movement. His name is James Laidler, and he was one of the ones who introduced me to this topic which I’ve written about many times both at my not-so-super-secret other blog and, of course, right here on SBM. In doing so, over the years I’ve catalogued why “autism biomed” seems compelling to many parents with children with autism, how antivaccine groups use fake “medical conferences” to sell autism biomed by giving a patina of medical respectability to rank quackery like bleach enemas, and providing a place where those selling unscientific treatments can find willing customers and where disreputable discredited “scientists” like Andrew Wakefield and Mark Geier can find adoring fans who believe their quackery.
I bring this up because last week WIRED published an excellent article about Jim Laidler, “An Alternative-Medicine Believer’s Journey Back to Science.” Appropriately enough, it’s by Alan Levinovitz, an assistant professor of religion at James Madison University. I say “appropriately enough” because, as has been noted here on a number of occasions, there are many religion-like aspects to alternative medicine in general but to the autism biomed movement in particular. Indeed, the two are often tied together, with the motivation for some alt-med being explicitly religious and belief in alt-med sharing some major characteristics with religion, particularly belief in miracles against evidence, charismatic leaders (like Andrew Wakefield) who can do no wrong, and mutually-supportive communities of believers who reinforce each others’ beliefs and ward off skepticism. Add to that the magical thinking, and it’s not for nothing that I’ve referred to the central dogma of alternative medicine as being that wishing makes it so. Indeed, it’s for good reason that I frequently point out that most “energy medicine” (particularly reiki) is basically faith healing that substitutes Eastern mysticism for Judeo-Christian religious beliefs.
In Levinovitz’s profile of Jim Laidler, we see a lot of this, and I learned some details that I didn’t know about Laidler before. Levinovitz also grasps the religion-like nature of alt-med by starting the article bluntly saying:
Jim and Louise Laidler lost their faith on a trip to Disneyland in 2002, while having breakfast in Goofy’s Kitchen.
Jim Laidler and his wife Louise are both doctors, and two of their sons, Ben and David, were diagnosed with autism. In the late 1990s until what happened in 2002, the Laidlers were very much true believers in autism biomed. I knew this back in the day, because I had read Jim Laidler’s writing extensively before; indeed, it was his recounting of his experiences treating his sons with autism biomed quackery that helped me to understand how confirmation bias, regression to the mean, placebo effects, and observer effects could lead parents to believe that pseudoscientific treatments were working to “recover” their children from autism when they were doing nothing to impact the course of the condition. He was the first one to teach me that, contrary to the myth that leads parents to attribute any improvement in autistic children to whatever quackery du jour they are using, autism spectrum disorders are conditions that result in neurodevelopmental delay, not neurodevelopmental stasis. In fact, autistic children can and do recover to the point where they “lose the ASD diagnosis” in anywhere from 3% to 25% of cases. (Yes, the literature is a variable on this point.)
So, there they were, the Laidlers, treating Ben and David with a wide variety of quackery to “recover” them from autism, as recounted by Levinovitz:
For several years, on the advice of doctors and parents, the Laidlers treated their children with a wide range of alternative medicine techniques designed to stem or even reverse autistic symptoms. They gave their boys regular supplements of vitamin B12, magnesium, and dimethylglycine. They kept David’s diet free of gluten and casein, heeding the advice of experts who warned that even the smallest bit of gluten would cause severe regression. They administered intravenous infusions of secretin, said to have astonishing therapeutic effects for a high percentage of autistic children.
Using substances known as chelating agents, the Laidlers also worked to rid Ben and David of heavy metals thought to be accumulated through vaccines and environmental pollutants. With a PhD in biology as well as his MD, Jim Laidler had become an expert on chelation, speaking nationally and internationally about it at conferences dedicated to autism and alternative approaches.
I’ve discussed such treatments many times before here at SBM, particularly chelation therapy, which has no basis in science for treating autism and whose rationale is predicated on the belief that the mercury in the thimerosal preservative that used to be used in many childhood vaccines causes autism. It’s a dangerous treatment that has caused at least one death. Autism biomed in general can be viewed as uncontrolled and unethical experimentation on a vulnerable population, autistic children. It’s definitely worth pointing out that, with an MD and a PhD—just as I have!—Laidler had the intellectual and scientific background to recognize that what he was doing was without a basis in science and medicine. Yet he didn’t, at least not for several years. Indeed, he helped promote chelation therapy and “spread the good news,” so to speak. How did this come about?
The journey to belief
In Levinovitz’s account, the Laidlers’ journey began after their eldest son, Ben, was born. He didn’t hit his milestones on time, such as rolling over, crawling, and walking. By the time he was three and a half years old, his speech consisted largely of echoing, and the Laidlers could no longer attribute his developmental delays to being at the low end of the bell-shaped curve for normal variation in development. In fact, by this time, Laidler was beginning to suspect, based on Ben’s lack of eye contact and other behaviors, what was probably going on, recounting it thusly, “I knew what this looked like. But I didn’t want to bring it up, not even to Louise. You get superstitious, like if you say something it’ll come true.”
Their son David was born three years after Ben, and initially he hit all his developmental milestones early. The family moved to Portland, and the Laidlers discovered what then made up the autism biomed movement on the Internet, mainly online forums where parents traded success stories. Remember, this was 1997, long before the many, many antivaccine, pro-autism biomed websites that exist now, such as Age of Autism, The Thinking Moms’ Revolution, and the like. Andrew Wakefield’s infamous and now retracted Lancet case series wouldn’t arrive until next year, and the full fear mongering over mercury in vaccines was still a few years away. For example, Robert F. Kennedy, Jr. didn’t publish “Deadly Immunity” until 2005. Jenny McCarthy’s antivaccine activism wouldn’t begin until a decade later. To bring it back around to how I started this post, there was Usenet, and there were private discussion forums, and that was about it. And, yes, I do recall reading Laidler’s description of how those forums were used mainly for parents to share “success stories” that, when he later looked at them critically, were not. No quackery was too outrageous, and all had validity.
Unfortunately, after hitting his milestones early for a while, David regressed. It was not long after that before he was diagnosed with autism as well. It was at this point, not surprisingly and in a very human reaction, the Laidlers fell into a depression.
And then:
Then another parent told Louise about Defeat Autism Now! She decided to attend.
The conference was a festival of alternative medicine, a campground of tents and booths and speakers offering attendees everything from cutting-edge supplements to dolphin therapy. Everyone—parents and presenters alike—swore by their preferred treatment, and countless stories circulated about severely autistic children who instantly turned around on the right protocol.
Louise returned filled with new hopes. At first, Jim was skeptical—”I hadn’t been there. I hadn’t felt the excitement”—but soon, he was hopeful too. They decided to put David and Ben on more supplements. They tried secretin infusions. They attended another DAN! conference. They came to believe in many other alternative therapies.
DAN!, as you might recall, consisted of a large number of practitioners offering all manner of quackery under the rubric of “autism biomed.” These doctors were listed on the registry of the antivaccine autism biomed group known as the “Autism Research Institute” an organization founded by Dr. Bernard Rimland in 1967. DAN! physicians generally became DAN! physicians by undergoing training through the ARI, which held yearly DAN! conferences to discuss the latest autism biomed quackery. Interestingly enough, in either an apparent attempt to become more “legitimate” or in response to lawsuits against DAN! doctors, ARI stopped maintaining its DAN! registry at the end of 2011. Back in the late 1990s through much of the first decade of the 21st century, though, if you wanted to find an autism biomed doc, the DAN! list was the first place to go. Unfortunately, many parents, like the Laidlers, did just that.
The Laidlers, however, eventually realized their mistake. Many parents, such as those who populate the antivaccine crank blog Age of Autism as bloggers or the inappropriately named “Thinking Moms” of The Thinking Moms’ Revolution never do. So what “converted” the Laidlers back to science? To understand this, we have to go back to what happened in Goofy’s Kitchen thirteen years ago.

The deconversion
So what, exactly, happened in Goofy’s Kitchen at Disneyland? It’s something that to most people would seem relatively minor but to the Laidlers immediately took on outsized significance. However, it’s also a good example of how a believer’s mind needs to be “primed” before an incident like this can tip him or her back towards skepticism. Before this event, Jim Laidler had started to doubt the whole DAN! subculture and what it was preaching. Levinovitz recounts how Laidler had begun to cringe when he heard of other parents mortgaging their homes to pay for expensive unproven autism biomed treatments and how “alarms went off” when parents and doctors would advocate dangerous protocols, such as extreme forms of chelation. So he spoke out against the more dangerous and extreme treatments being promoted and offered by DAN! doctors. That’s when he started to learn that skepticism is not appreciated in movements like DAN! Indeed, a prominent conference organizer took him to task for such criticisms, warning him “never to criticize anyone’s approach, no matter how crazy or dangerous it seemed.” We see a less extreme form of this same attitude in the seemingly more “reasonable” world of “complementary and alternative medicine” (CAM) or “integrative medicine,” where, seemingly, every hypothesis and treatment is potentially valid, even homeopathy, and CAM advocates are loathe ever to dismiss any treatment, even when it’s magical faith healing like reiki or sympathetic magic like homeopathy. More on that later.
First, let’s see what happened at Goofy’s Kitchen:
It was in the grip of these doubts when, inside Goofy’s Kitchen, Jim and Louise returned to their table from the buffet and noticed 6-year-old David hadn’t come with them. They saw him standing at the buffet, devouring a waffle. The Laidlers feared the worst. “We’d been told that the slightest smidgen of gluten would crash him,” Jim says. “It was absolutely devastating to watch.” But by the end of the vacation, they realized David was fine. Nothing happened.
When they returned home, the Laidlers took David off his restrictive diet, and he continued to improve—rapidly. Louise stopped Ben’s supplement regimen—without telling Jim—and Ben’s behavior remained the same. Then, after months of soul-searching, Jim Laider took to the internet to announce his “de-conversion” from alternative medicine—a kind of penance, but also a warning to others. “I had this guilt to expunge,” Jim says. “I helped to promote this nonsense, and I didn’t want other people to fall for it like I did.”
Jim Laidler had become an alt-med apostate. As anyone who’s studied religion knows, in general believers detest apostates far more than unbelievers. After all, unbelievers might still be made into believers and might just be ignorant of the belief system of the religion. Apostates, on the other hand, were once believers who were fully invested in belief but then actively decided to reject the religion. In this, he is like Britt Hermes, who was a naturopath but abandoned the faith after recognizing the pseudoscience at its heart, or Sam Homola, a chiropractor who rejected chiropractic’s more outlandish claims for the same reasons, or Paul Ingraham, a former registered massage therapist who couldn’t bear the pseudoscience anymore. After he announced his deconversion, Laidler suffered the usual fate of the apostate, shunning and attacks:
When Jim Laidler became an alt-medicine apostate, proponents of the treatments he criticized went on the attack. He received death threats from parents. It’s an intellectually and emotionally bruising battle no matter which side you’re on, one that today pits not just doctor against layperson, but also doctor against doctor. And as the Laidlers demonstrate, it can pit well-trained doctors against their own psyche. “I was happier because we felt like we were doing something right,” Jim Laidler says of the treatments he gave his sons over the course of so many years. “That’s how the madness begins. You want to believe that it’s working, so you force yourself to see results, and silence the scientific part of your brain.”
I know that Laidlers put up with such attacks for years. He had been fairly quiet over the last few years, which makes me wonder why he agreed to be the subject of a story published on a major website today, given that no doubt the reminder of his apostasy, correctly described as apostasy, will attract the attention of the autism biomed believers with whom he used to worship at the church of chelation. Whatever his reasons, I’m glad he did agree. I only wish Levinovitz had contacted me, because one thing that’s lacking in his piece is an understanding of just how influential Jim Laidler has been to a generation of skeptics (including myself) trying to counter the misinformation of the antivaccine movement and how useful his writings have been to us.
It has been said that alt-med appeals to some unmet need that scientific medicine either does not meet or does a poor job of handling. Whether that is a need for hope when there is little or none or the feeling that “something” is being done, that is perhaps true in many cases. However, unmet psychological needs are not a justification for the acceptance of pseudoscience creeping into conventional medicine, something we at SBM have been warning about for over seven years now. The story of the Laidlers shows how even doctors and scientists can be ensnared as patients and parents of patients, but what about the providers?
The return of David Katz and his “more fluid concept of evidence”
Four weeks ago, I discussed the unfortunate case of Dr. David Katz, who happens to be faculty at Yale along with our very own Dr. Steve Novella. Dr. Katz, you might recall, first found himself on the radar of skeptics when he advocated a “more fluid concept of evidence” with respect to medicine back in a now infamous 2008 talk. His message has been correctly characterized as, “Don’t abandon patients; abandon science instead.” Well who to my wondering eyes should appear in Levinovitz’s story but Dr. “More Fluid Concept of Evidence” himself? His message is very revealing, albeit perhaps inadvertently. After noting quite correctly that twenty years ago, when the Laidlers’ journey into alternative medicine began, what they sought for their sons was clearly outside the medical mainstream, “largely separate from mainstream hospitals and medical schools,” as he put it, Levinovitz notes that, thanks to what is now known as “integrative medicine,” such is no longer the case:
At Yale, physician and prominent health advocate David Katz practices integrative medicine, aiming to combine acceptance of alternative treatments with scientific rigor and regulation. Katz opposes chelation, along with all non-standard protocols proven to be physically harmful, but he supports being open-minded about many other alternative techniques, including acupuncture, homeopathy, and reiki.
Such open-mindedness, he says, offers a badly needed corrective to a broken medical system. “With internal medicine, once I’ve tried everything the textbooks tell me, I’m done,” he says. “But with integrative medicine, I always have something to try. I never run out of options.”
This argument is atrociously dumb, even by David Katz standards. Yes, it might be true that with integrative medicine a physician “never runs out of options,” but if those options are fantasy and fairy dust, they do the patient no more good than “throwing up one’s hands” and saying there’s nothing that can be done. Let’s put it this way. If magic like homeopathy and reiki (which, recall, is nothing more than “faith healing”) are “options,” then where does it end? How far outside the realm of well-established science does a proposed treatment have to be before Katz will reject it as not worth considering as an “option”? That he has embraced homeopathy and reiki tells me that there is probably no treatment so outrageously ridiculous that he won’t consider it.
It’s also not true that “once you’ve tried everything in the textbooks,” you’re done. If that’s the limit of your thinking, such that it either has to be in the textbook, or you’ll start considering fairy dust, then it’s your thinking that is limited, not that of skeptics. Just ask Mark Crislip, who just provided us suggestions for what to do when the evidence runs thin. In medicine, unfortunately, sometimes all the options are bad, and it’s a matter of picking the “least bad” option. Some things can’t be cured, but they can almost always at least be palliated. As is pointed out elsewhere in the WIRED article, expectation management is an important part of medicine, and it is there that medicine frequently falls down. Integrative medicine lets doctors off the hook. They no longer have to manage expectations, or at least they can give patients hope that they wouldn’t otherwise be able to give. It doesn’t always matter if it’s false hope:
And false hope isn’t just for patients. Doctors—traditional and integrative alike—also can become dependent. No doctor wants to be hope’s executioner, especially when they don’t know how to do it properly. The vast majority of doctors receive no formal training in how to deliver catastrophic news. Most hospitals don’t provide a social worker to take over once parents find out their child has autism. Instead of thinking about what to do when there are no options for a cure, it’s easier, as Katz puts it, to never run out of options.
No doctor wants to be hope’s executioner. How true. I’ve had to tell patients’ families in the past of dire circumstances with little or no hope. I’ve had to help families come to a decision to withdraw support. I’ve even had to be the one to withdraw support. I’ve had to tell patients and their families that there was nothing more that could be done to save their life. Yes, these were all a long time ago, before I hyperspecialized in breast cancer surgery, where, fortunately, such conversations are now fairly rare, but I remember such incidents clearly because of how hard they were, mentally, intellectually, and emotionally. They were painful. But they were part of my job, my profession. Giving patients the hard news was my duty. How much easier it would have been if I could have come up with “options” that would prevent me from having to present just the cold, hard, facts and prognosis to patients and their families.
As our fearless leader, Steve Novella, who was quoted in Levinovitz’s story, relates:
When confronted with the possibility of a troubling diagnosis, people often prefer to avoid hard truths. Novella tells a gut-wrenching story about one of his patients, who had ALS. After the initial diagnosis, the man left the office, unable to cope with the fact that he had an incurable degenerative disease, likely to kill him within five years. He chose to visit a naturopath instead, who had redemptive news: conventional, narrow-minded medicine had misdiagnosed him. It wasn’t ALS, the naturopath said, but rather chronic Lyme disease, which could be treated with holistic, all-natural supplements.
Nearly a year later, badly degenerated, the man was back in Novella’s office. He’d wasted countless hours and thousands of dollars on false hope. Now, he was willing to listen, but with far less time to prepare for the reality of what lay ahead, and a spirit broken by disappointment. Research suggests this is no mere anecdote. Studies out of Norway, Japan, and Korea have reported higher mortality rates and lower quality of life for cancer patients who pursue complementary and alternative medicine.
Whether he knows it or not, perhaps Dr. Katz has inadvertently revealed the probable primary motivation for the “integration” of quackery into conventional medicine, although not quite in the way he thinks he did. Reading this line of argument, I can’t help but conclude that the embrace of more “options” in integrative medicine is not so much about the patient, Katz’s protestations otherwise. Think about it. Katz argues that integrative medicine means “never running out of options.” While there’s no doubt that options can be a good thing when they are options not based on fairy dust that might actually do something, by embracing prescientific, pseudoscientific, and outright magical “options,” integrative medicine helps doctors avoid making the hard choices that their patients need them to make, or at least helps them put off those choices, often for long periods of time. That’s what it’s really about, not doing whatever it takes to make patients feel better. All that this requires is for physicians to be so open-minded that their brains are allowed to fall out.
Yes, there are problems with conventional medicine. Doctors often spend too little time with patients because the reimbursement system requires that they see lots of patients to be able to make a living. The medical system often doesn’t deal with terminal illnesses or chronic illnesses as well as it should. However, to paraphrase Ben Goldacre, just because there are problems with aircraft design, that doesn’t mean magic carpets really fly, and just because there are problems in our medical system does not mean that the solution is to embrace pseudoscience and magic. Jim Laidler was one of my early skeptical influences who helped teach me that. Ironically, David Katz is also teaching me the same thing, but mainly as a negative example.
The road to apostasy, as Jim Laidler shows us, is hard. That’s why it’s important to do what we can to prevent conversion to pseudoscience in the first place. Unfortunately, that is often not possible. Even more unfortunately, David Katz and other advocates of integrating quackery with medicine show that deconverting doctors who have embraced integrative medicine will likely be just as difficult. At least Dr. Laidler shows us that it can happen. Can it happen with doctors like David Katz? I hope so.

Attitudes Predict CAM Use
Steven Novella on May 27, 2015
Link

One of the persistent themes of SBM is that CAM (complementary and alternative medicine, or integrative medicine) is nothing more than a marketing brand. Its recent popularity is not based upon new evidence or a changing paradigm of medicine as its proponents claim. Its popularity is increasing despite the lack of evidence for specific CAM treatments and despite a dedication to evidence-based medicine within the medical profession.
CAM is also modern mythology, which I guess all really effective advertising and branding is. It floats atop a number of demonstrably false marketing claims. One is that the popularity and use of CAM is surging. This is partly a self-fulfilling prophesy, and no doubt it is increasing, but the degree to which CAM is popular has been consistently exaggerated by proponents (largely as a way to justify its existence).
This myth is largely perpetuated by redefining CAM as needed, including things like prayer, massage, and taking vitamins. I suspect that praying for a sick loved-one has always been popular and doesn’t represent a trend toward CAM. When unequivocal alternative modalities are considered, their use is still tiny and not increasing. The most recent NIH survey found:
Use of acupuncture (1.1%), homeopathic treatment (1.7%) naturopathy (0.2%), and energy healing (0.5%) was miniscule.
Other myths of CAM marketing include that the treatments are “natural,” that they are entirely safe, and in many cases that they are ancient. Collectively these claims create an emotionally compelling narrative or image – we can almost see the “noble savage,” one with nature, ancient wisdom evident in their gentle gaze, as they lay their healing hands on the afflicted. Imagine a boardroom scene from Mad Men as they pitch this idea.
Perhaps the biggest CAM myth, however, is the notion that people turn to CAM primarily because they are dissatisfied with mainstream medicine. I think this is partly the default assumption that most people make, but it is also perpetuated by CAM proponents. So popular is this myth that I find it is common even among CAM critics and proponents of SBM.
The problem with this assumption is that it is at odds with the actual evidence that we have. Studies show that satisfaction with mainstream medicine is not an important factor in deciding to use CAM, that CAM users are generally satisfied with their mainstream care, and they use CAM because it aligns with their philosophy, and they simply want to expand their options.
A 2007 review found that people who seek CAM hold a number of beliefs and motivations. These include a desire for control, a belief in “natural” remedies and “holistic” medicine, and a belief that psychological factors are important to health.
Overall world view is also highly predictive of CAM use. Specifically, a world view researchers have called the “cultural creatives”:
This group is said to represent unconventionality and is characterized by commitment to causes such as feminism, environmentalism, spirituality, personal growth and a love of the foreign and exotic.
The evidence also suggests that the same factors are involved in maintaining CAM treatment. While CAM practitioners are quick to cite the alleged effectiveness of their treatments as the reason their customers keep coming back, CAM users cite empowerment, communication, and beliefs about treatment as their main reasons – even when they perceive that the treatment is not working. They also do not cite dissatisfaction with conventional medicine.
A 2015 survey of CAM use among cancer patients had similar findings. They found that attitudes toward CAM are what predicted CAM use. Positive attitudes toward CAM also correlated with being young, female, and college educated, factors which also correlate with the “cultural creatives.”
Conclusion
The evidence clearly shows that CAM is a cultural phenomenon, not a result of change in scientific thinking, and not an emergence of new evidence. It is a brand that taps into a particular subculture, one that values the natural, ancient, and exotic. Surveys also show that CAM use has leveled off in the last 10 years, perhaps because it has reached full market penetration in the target demographic.
However, there is still significant spill-over into the general culture. The dedicated minority of CAM enthusiasts have been successful in spreading their brand, and infiltrating specific markets, such as hospitals, universities, and pharmacies. They have been politically successful as well, securing dedicated research funding (which in practice is just another method of marketing) and favorable regulations.
They have also, quite disappointingly, put academics off their game. Most academics are completely unfamiliar with the CAM phenomenon and give it little thought (what we call “shruggies”). A few see it for what it is (an attack on the scientific basis of medicine) but feel constrained by the forces of political correctness, which are powerful within the halls of academia. This is a generational failure. How this is looked upon in the future will likely depend upon which philosophical outlook emerges as dominant.
At the very least we need to expose the facts as they are. The facts tell a very different story from the CAM marketing mythology.

How We Believe
James Alcock’s new book about belief is a masterpiece that explains how our minds work, how we form beliefs, and why they are so powerful. It amounts to a course in psychology and an owner’s manual for the brain.
Harriet Hall on June 26, 2018
Link

In James Alcock’s classic 1995 article “The Belief Engine“, he said, “Our brains and nervous systems constitute a belief-generating machine, a system that evolved to assure not truth, logic, and reason, but survival.” Now he has expanded that thesis into a book, Belief: What It Means to Believe and Why Our Convictions Are So Compelling. It’s much more than a book about belief. In the Foreword, Ray Hyman says it would be an ideal textbook for a course that provides an integrated overview of all the areas of psychology. He says every psychologist and psychology student should read it. It is an outstanding achievement of scholarship; its 640 pages include over 70 pages of references. It covers everything from the latest findings in neuroscience to a catalog of many of the questionable beliefs people hold, and why they hold them.
Eminently qualified
Alcock is the ideal person to write such a book. He has a BSc in physics and a PhD in psychology and has been teaching psychology at York University since 1973. He is one of the founders of the Committee for Skeptical Inquiry (formerly CSICOP), a CSI Fellow and member of its Executive Council, and has won numerous awards for skepticism and psychology. He has written extensively on social psychology and the psychology of belief. He is also a registered clinical psychologist with his own private practice, and also an amateur magician. There’s an extensive article about him on Wikipedia. He has stature in both senses of the word: (if I remember correctly he’s 6’7″).
The power of belief
Beliefs guide all our thoughts and behaviors, from brushing our teeth to voting for a particular political party. They have power over life and death: people have willingly died for their beliefs, and someone commits suicide every forty seconds. Alcock elucidates the various factors that contribute to suicide terrorism. And he tries to explain why some beliefs are so powerful that they are impervious to reason and evidence.
The belief engine
Alcock says there is nothing fundamentally different about the nature of beliefs that we consider rational and those we deem irrational. We do not choose our beliefs; they are generated and maintained through automatic processes in our brains. He explains what goes into those automatic processes: perceiving, remembering, learning, feeling, and thinking. And he shows how those processes can depart from reality.
Perceiving
The brain uses sensory input to construct schemas that may not represent the real world accurately. It fills in missing information. It creates visual illusions. Attention is selective: we think we are aware of everything in our environment; but we aren’t. We see things (pareidolia) and hear things (apparent words in random noise) that are not really there. We sometimes confuse mental imagery with external reality. So, we need to be cautious when basing a belief entirely on what our senses tell us.
Memory
Recent research has revealed how unreliable our memory is, even when we are most confident that we are remembering correctly. Memories are reconstructed each time we remember. Experiments have shown how easily false memories can be implanted and elaborated. Contaminating influences can distort our memories in various ways. Memories “recovered” under hypnosis are confabulations. Eyewitness testimony is notoriously unreliable. Errors and bias in memory have led to false convictions and ruined lives. So, memories should not be treated as unshakeable foundations for our beliefs.
Learning and feeling
Alcock explains how we learn through our own experience, through watching others, through what we are taught, and through what we read. Many beliefs are established in childhood: children soak up new information like a sponge. He covers classical conditioning, operant conditioning, reinforcement, superstitious conditioning, and the power of coincidence. He shows that we must trust others for most of our information, and discusses how we decide whom to trust. He shows how beliefs influence emotions and emotions influence beliefs. Belief in magical remedies can reduce despair when scientific medicine can’t provide a cure. Learning from others means we must trust their accuracy, reliability, and honesty, which leaves us open to error and manipulation. We must use critical thinking skills to help us separate fact from fiction.
Thinking processes
He explains the two types of thinking: System 1 (experiential, intuitive) and System 2 (rational). System 1 thinking involves intuitions and rules of thumb that are a practical necessity for rapid response and the necessities of daily life. But our intuitions are subject to many biases such as the availability heuristic, the simulation heuristic, the representativeness heuristic, the introspection illusion, and confirmation bias. He discusses hunches gone wild, the Gambler’s Fallacy, misunderstanding of probability and statistics, the need to look at base rates, etc. System 2, rational processing, can be led astray by errors of logic, belief bias, abduction reasoning, and enthymematic reasoning. Logic doesn’t come naturally, and it can be trumped by emotion and intuition and be sullied by various errors and biases. It’s important to remember that even rational thinking can be fallible.
Belief stability and change
In this section, Alcock focuses on how beliefs are formed, how some beliefs resist change while other submit to contradictory information, and how beliefs motivate people to achieve difficult goals, even to the extent of dying for them. We all hold beliefs that are wrong but that seem just as reasonable as any other belief. We automatically believe new information, and only later do we examine it for truth. True knowledge must be based on evidence, but we can’t personally verify many of our most important beliefs, so we must rely on the pronouncements of authorities.
Before people’s actions and allegiances change, their beliefs must change. Many beliefs are so entrenched that they resist powerful evidence against them. He addresses collective delusions, conspiracy theories, and “moral panics” like the one that led to false accusations of Satanic ritual abuse. He lists the factors that make beliefs unlikely to change, the ways people rationalize away new evidence, and how disconfirmation can actually strengthen beliefs. And he discusses how sometimes even the most extreme beliefs can change through conversion. He covers persuasion, Hitler’s cardinal rules for successful propaganda, gas-lighting, brainwashing, interrogation, fake news, alternative facts, and post-truth. He describes some of the suggestions in the psychological literature for getting people to change their beliefs.
Credulity and deceit
He discusses suggestibility, the Barnum Effect, cold readings, imposters, con artists, hoaxes, Ponzi schemes, Nigerian email scams, lie detection, and self-deception. He stresses that we can all be fooled. We are poor at detecting deceit in others and we fail to recognize when we have fooled ourselves.
Self-knowledge
Alcock explains how our beliefs about our bodies, our minds, and our well-being sometimes stray significantly from reality. He covers various illusions like phantom limbs, the rubber hand illusion, body maps, the “ghost in the machine” belief, the mystery of consciousness, the workings of the unconscious, ideomotor actions (dowsing, Ouija boards, etc.). Recent research has established that our actions are determined unconsciously before we are consciously aware that we have decided to act.
Altered states of consciousness
The brain can fool us with remarkable experiences that we can’t distinguish from external reality, often accompanied by strong emotions and lasting effects. He covers transcendental experiences, hallucinations, out-of-body experiences, meditation, hypnosis, dreams, and sleep paralysis.
Belief and well-being
There can be illness (subjective symptoms) without disease (pathophysiology). Beliefs about the state of our health may not reflect the actual state of health but can contribute to it. Is stress harmful? The belief that stress is bad for us can be deleterious to our health. Alcock examines possibly unreliable reports of people scared to death, dying after hexes, the “broken heart” syndrome, etc. He discusses hysteria, mass hysteria, hypochondria, the worried well, and questionable diagnoses like multiple chemical sensitivity and electromagnetic hypersensitivity.
Belief and healing
Feeling better after a treatment doesn’t necessarily mean we actually are better. Suggestion is powerful, healing rituals are persuasive. He covers Mesmer’s “animal magnetism,” placebo effects, sham surgeries, learned responses, expectancy effects, conditioning, social learning, and theological placebos. He says there are three types of healing: natural healing (the body heals itself), technological healing (drugs, surgery) and interpersonal healing that depends on context and personal interactions and that leads to improvements in illnessbut not in disease.
Folk remedies and alternative medicine
He includes a long chapter on belief in remedies that either lack evidence of effectiveness or have been proven ineffective. He covers traditional Chinese medicine, qi gong, acupuncture, homeopathy, naturopathy, chiropractic, aromatherapy, therapeutic touch, Reiki, thought-field therapy, and cranial-sacral stimulation. He investigates the reasons people choose alternative medicine and reject science-based treatments and vaccines. He covers beliefs in psychology, including recovered memories, EMDR, Munchausen by proxy, and seasonal affective disorder. He concludes:
Let careful scientific research rather than hyperbole, gushing testimonials, or ‘ancient wisdom’ be the foundation for our beliefs about what is or is not effective.”
That would be a good motto for the Science-Based Medicine website.
Belief in a world beyond
We are born magical thinkers, and magical thinking is hard to overcome. Alcock delves into beliefs associated with magic, religion, superstition, and the paranormal, examining how they reflect our constructed representations of reality. He discusses stage magic, sympathetic magic, magical thinking, hyperactive agency detection, magical contagion, and the persistence of superstitions.
In separate chapters, he covers the origins and psychological benefits and harms of religion, anomalous experiences and parapsychology, illusory experiences, and a caboodle of strange beliefs, from alien abductions to the Bermuda triangle, from electronic voice phenomena to reincarnation, from fire-walking to energy fields, from spontaneous human combustion to astrology.
Building a firewall against folly
In a final chapter, he shows how we can all become better at critical thinking. He gives these eight rules:
     1.  Remember that we can all be fooled.
     2.  Be wary of your intuitions.
     3.  Be wary of the Fundamental Attribution Error, attributing people’s behavior to their characters and intentions while overlooking the power of the situation.
     4.  Be wary of validation by personal experience.
     5.  Don’t rely on a single source of information.
     6.  Don’t over-interpret correlations.
     7.  Ask “compared to what?” – a wine was rejected because it was found to contain two million asbestos particles per liter, but the concentration of asbestos particles in the city water supply was higher than that.
     8.  In the face of inadequate evidence, suspend judgment rather than jumping to conclusions.
Finally, he reminds us that critical thinking means we should be prepared to disagree with ourselves, which is never easy.
Conclusion: A great book
The book covers a huge variety of topics. It’s written in a style that is accessible and appealing to the lay audience yet rigorous enough to satisfy professionals. I think everyone would benefit from reading it. It is the equivalent of a psychology course and an owner’s manual for the brain; it explains how our minds work, how we come to believe the things we do and why it is hard to change those beliefs; it explains our biases and errors, and how critical thinking can help us distinguish true beliefs from false ones. It combines the latest scientific knowledge with the best incisive thinking. It provides insight into many of the problems facing our society. And it’s an entertaining encyclopedia of strange and false beliefs.
The book’s principles are illustrated by many fascinating examples and anecdotes. My favorite was the rectal piano. One of Alcock’s patients held the delusion that there was a small piano in his rectum, and he was obsessed with a desire to play it. He wanted to have his fingers surgically removed to eliminate the temptation! Alcock asked if he had considered having the piano removed; he thought that was a great idea and asked Alcock if he could do the surgery.
I can’t praise this book enough. You can borrow it from your public library for free or buy the Kindle edition for only $11.99. Read it! It will educate you and entertain you, and you will begin to question some of your beliefs that you might not have thought to question before. Our society would be a much better place if everyone would read this book and absorb its lessons.

Agnotology: The Study of Ignorance
Mark Crislip on March 21, 2014
Link

A comment from the blog:
Every single time – bar none – I have had a conversation with someone about CAM and its modalities, they are absolutely astonished when I explain to them what the modality really is. One story I love telling comes from my friend in the year behind me. His parents are professional chemists and he came home one day and saw his mother had a bottle of homeopathic medicine. He asked why and she gave the typical non-committal response of “well, I thought it may help and I saw it on the shelf at the pharmacy.” He explained what homeopathy actually is and they were absolutely dumbfounded. They are well aware of Avogadro’s number, after all. People generally don’t study what the CAM in question actually is – merely the fluff PR garbage that gets touted around and without direct and clear demonstration of harm, give it a pass as a result. After all, the business of real medicine is time consuming and difficult enough.
Participating in activities that have a permanent record gives one the fortunate, or unfortunate, opportunity to revisit the past and see just how you worked early in a career.
It was sobering, as third year resident, to see the notes I had written as an intern. Man. It was amazing how unsophisticated my medical thinking had been a little over two years’ prior. How little I understood about the ins and outs of diagnosis and treatment even after four years of medical school. It is part of the reason I think it is a joke, albeit a cruel joke, that naturopaths and other pseudo-medical providers think they can function as primary care providers after a few years of alternative edjamacation.
I have a similar experience every now and then when I see the notes from early in my Infectious Disease practice, now heading into 24 years. Not quite as painful, but still remarkable in how much I didn’t know then. My ID podcast is an ongoing reminder of how much I still do not know. The last 34 years have been my personal linear acquisition of knowledge from the exponential production in the medical, and non-medical, world. Ignorance isn’t bliss, but an ever-expanding hole that can never be filled.
There is nothing wrong with ignorance per se. It depends on what you do with it. Ignorance can be a condition you can spend a lifetime attempting to overcome.
My early career in the world of SBM was defined by a remarkable naiveté. I thought people who used the various pseudo-medicines were simply ignorant, they lacked basic information about the topic and all I had to do was supply that information. They would read/listen to my explanation and think, ‘Oh. That’s how it works’ and move on. That is partly how medical training works. Once you learn how some therapy or procedure does or does not work, you behave accordingly. Eventually. Change is painful, and I do notice as I age how much harder it is to make an intellectual change. Habit is so comfortable.
There was certainly a large helping of ignorance with a side of hubris on my part. But that is how you increase knowledge. Discover the gaps and fill them.
I have not thought much about ignorance. Most of my time is spent on the hows and whys of the acquisition of knowledge. Part of my job and my hobby is to be an educator. I think of residents, myself, and my readers as an empty glass to be filled with facts and their relationships. A simplistic idea, but how I spend a huge amount of my time. I take in information, organize it, synthesize it, and then pass it to others, hopefully in a clever manner.
Others have thought about ignorance in a more comprehensive way. Robert N. Proctor is such a person and he has coined a term for the cultural production (and study) of ignorance: Agnotology. The author delineates several kinds of ignorance in the paper, and they make for a good conceptual framework for understanding ignorance.
And though distinctions such as these are somewhat arbitrary, I shall make three to begin the discussion: ignorance as native state (or resource), ignorance as lost realm (or selective choice), and ignorance as a deliberately engineered and strategic ploy (or active construct).
And he points out that:
Ignorance has many interesting surrogates and overlaps in myriad ways with—as it is generated by—secrecy, stupidity, apathy, censorship, disinformation, faith, and forgetfulness,
I have tended to think of ignorance only in the first definition: simply lacking knowledge on a topic or having wrong or incomplete knowledge. No one can know everything or anything perfectly. It is the kind of ignorance I thought I was combatting when I started my blogging career and why I spend an inordinate amount of time on Pubmed.
It leads to the second kind of ignorance, that of selective choice. Due to time and interest (not all topics in the universe are equally interesting) there are areas about which I choose to have a minimal knowledge. Diet, I admit, bores me. I pay little attention to the ongoing debates as to the best diet and what is good and mad to eat. I eat for pleasure or fuel and not for health and it is not a part of my professional life.
Other examples of selective ignorance occur in professional education. Medical schools and residency give short shrift to pseudo-medicine and critical thinking, and probably justifiable so. There is just so much time and neuronal space for the jaw-droppingly huge amount of information that becoming a physician requires. I have mentioned before that I was in medical school and training from 1980 to 1990. That decade of my life was spent learning my profession. My kids used to like to watch I Love the 80’s on VH1 and I recognized nothing from the show: the movies, the music, the fashion, the memes were are new to me. Most people lose a decade of life to drugs or alcohol; mine was to medicine. I have a huge selective ignorance concerning the 80’s and if the show is any evidence, it was a good thing.
Even more impressive in their selective ignorance is the training in pseudo-medicines:
Ignorance is a product of inattention, and since we cannot study all things, some by necessity—almost all, in fact—must be left out. “A way of seeing is also a way of not seeing—a focus upon object A involves a neglect of object B.”
And best typified by the curricula at a naturopathic school, but the same is true for any pseudo-medical education. Vast quantities of time are spent on areas divorced from reality: homeopathy, acupuncture, hydrotherapy etc. The effort to absorb these fantasies is, by their nature, going to prevent acquisition of knowledge about reality.
There is a similar process occurring at chiropractic schools, where they like to brag about their education:
According to the American Chiropractic Association, the course of study to become a chiropractor includes 4,200 hours of classroom, laboratory and clinical experience in “orthopedics, neurology, physiology, human anatomy, clinical diagnosis including laboratory procedures, diagnostic imaging, exercise, nutrition rehabilitation and more.”
I am not impressed. That 4,200 hours is 525 eight-hour days, a little under a year and a half. It would be so much more impressive if they said their training was 15,120,000 seconds. The bigger the number the better the education. My internal medicine training was seven years, plus two more for infectious diseases. And what good is all that training if it is being applied to the fantastical ideas of fixing subluxations? It is like learning horse anatomy to take care of unicorns. But it also ensures ignorance in areas of reality-based medicine.
The most interesting form of ignorance is the third:
Ignorance as strategic ploy, or active construct
The focus here is on ignorance-or doubt or uncertainty-as something that is made, maintained, and manipulated by means of certain arts and sciences. The idea is one that easily lends itself to paranoia: namely, that certain people don’t want you to know certain things, or will actively work to organize doubt or uncertainty or misinformation to help maintain (your) ignorance. They know, and may or may not want you to know they know, but you are not to be privy to the secret. This is an idea insufficiently explored by philosophers, that ignorance should not be viewed as a simple omission or gap, but rather as an active production. Ignorance can be actively engineered part of a deliberate plan.
The author uses the tobacco industry as an archetype of an industry that manufactures ignorance and starts the paper with a quote:
Doubt is our product. Brown & Williamson Tobacco Company, internal memo, 1969
The manufacture of doubt is common in the pseudo-medical world. It could not exist without it. I do not know if ignorance is bliss for pseudo-medicine, but it is a requisite.
The false information that underlies all pseudo-medicine, from the popularity of pseudo-medicine in the US, to the efficacy of acupuncture to the safety of chiropractic to the mechanism of reiki relies on the production of massive amounts of ignorance. And so has some aspects of real medical treatments. Pharmaceutical companies have not been hesitant to borrow methods from their tobacco brethren. Although science can be an antidote to the production of ignorance in the real world, the pseudo-medical world is often invulnerable.
Examples of ignorance as strategic ploy in the pseudo-medical world abound and can, for a time be effective, as Megan Sandlin demonstrated, although:
In the end, I couldn’t continue to deny the science. It’s hard to believe now how easily I bought into everything I was hearing from the anti-vaccine crowd. It seems extremely obvious now: doctors aren’t evil, scientists aren’t trying to kill your kids with toxins, and vaccine researchers aren’t just trying to scam you out of your money.
Natural News and the Mercola site are probably the Ford and GM of medical ignorance production, but there are numerous boutique producers. I ran across Why You Never Need A Tetanus Vaccine, Regardless of Your Age or Location by Dave Mihalovic, ND, whose ignorance production I have discussed before
Mr. Mihalovic identifies himself as “a naturopathic medical doctor who specializes in vaccine research.” However, just where the research is published is uncertain as his name yields no publications on Pubmed. BTW. I specialize in beer research. Same credentials.
Tetanus is a rare disease in the US. I have seen one case, years ago as a fellow, in an elderly immigrant who had never received the vaccine. Having every muscle spasm at once is horrible for the victim.
From 1922-1926, there were an estimated 1,314 cases of tetanus per year in the U.S. In the late 1940’s, the tetanus vaccine was introduced, and tetanus became a disease that was officially counted and tracked by public health officials. In 2000, only 41 cases of tetanus were reported in the U.S. … Approximately 20 percent of reported cases end in death.
Tetanus in the U.S. is primarily a disease of adults, but unvaccinated children and infants of unvaccinated mothers are also at risk for tetanus and neonatal tetanus, respectively. From 1995-1997, 33 percent of reported cases of tetanus occurred among persons 60 years of age or older and 60 percent occurred in patients greater than 40 years of age. The National Health Interview Survey found that in 1995, only 36 percent of adults 65 or older had received a tetanus vaccination during the preceding 10 years.Worldwide, tetanus in newborn infants continues to be a huge problem. Every year tetanus kills 300,000 newborns and 30,000 birth mothers who were not properly vaccinated.
A rare, awful, and mostly preventable disease, it is caused by Clostridium tetani. The bacteria, found in the soil, gets into damaged tissues, releases its toxin and the result is tetanus.
Tetanus toxin, tetanospasmin, is extremely potent and can cause severe disease yet not cause the production of antibody. A curiosity of many toxins made by Clostridia, be it botulism, tetanus or gas gangrene, is the purpose of the toxins in the wild (i.e. the dirt), which remain a mystery. As an anaerobe it can be difficult to grow.
Of course, as is his métier, and no doubt a result of his naturopathic training where his understanding of microbiology and infectious diseases is profoundly ignorant, he says:
The tetanus bacteria may be a factor in tetanus. The toxin may be involved in some way but that these are fundamental causes is nonsense, otherwise the disease would be more common, in view of the fact that the bacteria is so frequently found on and in our bodies.
And:
The real cause of tetanus is not a germ, but dirt and filth. The bacteria are harmless when placed into a surgically clean wound. Tetanus develops when drainage of a wound is checked and dirt is retained in the tissues.
And:
The patient suffering from tetanus should be put to bed, permitted to rest, kept warm and fasting should be immediately instituted. They should receive all the salubrious hygienic influences and the fasting should be continued until all symptoms have disappeared.
Advice and treatment that, if followed, could result a repeat of when parents’ fear of vaccinations nearly killed their son or worse:
Auckland parents Ian and Linda Williams thought they had made an informed choice not to vaccinate their children, but after their son ended up in intensive care with a tetanus infection they realized they had made a terrible mistake.
The problem with reality is it doesn’t care if you are ignorant. You can reject and substitute your own where dirt causes tetanus and vaccines are worthless. Get the perfect storm of bad luck and you will get tetanus if not vaccinated.
Pseudo-medicine is producing ignorance at a vastly higher rate than medicine can produce an approximation of the truth. But it will always be that way. It is why I lobbied for Sisyphus to be the emblem of the Society for Science-Based Medicine.

Stupidity
His brain may hurt, but he ain’t stupid.
Mark Crislip on October 20, 2026
Link

“Against stupidity we are defenseless.”
    • Dietrich Bonhoeffer
I miss serendipity. The Al Gore rhythms (that is how it is spelled, right?) feed me much the same crap I saw before. Once you click on a link you are doomed to the Internet equivalent of Groundhog Day. I want to see something new and different, not another English bulldog clip, even if they are damn cute.
One of the fun aspects of doing searches on your own, instead of using a librarian or an LLM, is the odd article or topic that shows up out of right field that leads to going down an interesting rat hole.
Those off-the-wall, unexpected articles are now rare, as most of the big sites keep you in self-selected silos. I do not want this kind of random weird site, although some come close.
Years ago, I stumbled across agnotology, the study of ignorance. The patron Saint of RFK Jr. Or maybe that is Saint Upid.
And speaking of St. Upid, a video popped up in my feeds that, as best I can tell, had nothing to do with prior clicks and the Al Gore rhythms. A little sweet serendipity.
Did you know that stupidity is a philosophical concept? With a philosophical definition? And stupidity is more common than evil and far worse? Well, I didn’t. And since I journeyed down this rat hole, I thought I would drag you along with me.
On Stupidity was written in 1943 while the author, Dietrich Bonhoeffer, was awaiting execution for participating in the assassination attempt on Hitler. He was hanged in 1945. As Samual Johnson noted, “Depend upon it, sir, when a man knows he is to be hanged in a fortnight, it concentrates his mind wonderfully.”
I can only imagine how focused he was as he wrote the essay.
Bonhoeffer was concerned with why his fellow Germans went along with Hitler. Because they were stupid, not evil.
Stupidity here is not low IQ, LLM defined, or self-defined or ignorance.
As Bonhoeffer noted
There are human beings who are of remarkably agile intellect yet stupid, and others who are intellectually quite dull yet anything but stupid.
Stupidity is part of being human, a moral and sociological condition that has three components:
Surrender of Independence: The core of stupidity is a voluntary abdication of one’s own critical thinking, a willingness to surrender one’s inner freedom.
“…under certain circumstances, people are made stupid or that they allow this to happen to them.”
Group Conformity: This surrender is often driven by a desire to belong, to conform to a group, ideology, or leader.
“Upon closer observation, it becomes apparent that every strong upsurge of power in the public sphere, be it of a political or a religious nature, infects a large part of humankind with stupidity.”
Resistance to Facts: The stupid person actively resists facts and arguments that contradict their pre-conceived notions. They are immune to reason.
” The fact that the stupid person is often stubborn must not blind us to the fact that he is not independent. In conversation with him, one virtually feels that one is dealing not at all with him as a person, but with slogans, catchwords, and the like that have taken possession of him. He is under a spell, blinded, misused, and abused in his very being.”
Stupidity, he argues, is more dangerous than evil, as evil often has the seeds of its own end.
What is the cure for stupidity? Liberation of the mind with questioning and critical thinking, although Bonhoeffer notes,
It really will depend on whether those in power expect more from peoples’ stupidity than from their inner independence and wisdom.
Fortunately, this definition of stupidity is only of historical interest. We live in a post-stupid society, and the above worries no longer apply to current mores.
You only need to look at the Internet, the Congress, or the CDC to realize how far as a society we have moved on from stupidity. Certainly those in power do not promote stupidity. Such an approach would be catastrophic to the health and well-being of the nation. It would allow dangerous pseudoscience to spread through the body politic like, well, I don’t want to have this essay devolve into Godwin’s Law. Next thing you would know, we would have a measles outbreak or the idea that RSV was produced in a lab to infect people.
Why are we protected from stupidity? Because we are a highly literate society, filled with citizens who can and do spend their time reading. Reading is perhaps the only way to comprehend complex information and to understand the whys and wherefores of other people and ideas, people and ideas different from themselves. Imagine a different society other than the one we are fortunate to live in.
Imagine the consequences to critical thinking,
If people spent 4 plus hours a day doomscrolling on the phone?
If basic literacy was falling and more than 1 in 4 U.S. adults read at the lowest literacy level, up from about 1 in 5 in 2017?
If college freshmen were unable to read and had never read an entire book?
If only 30% of Gen Alpha fourth graders read proficiently as of 2024?
If people relied on LLMs instead of their brains?
If having a library card was a reason to dump your girlfriend?
If using large words was considered ableist? What would become of Mary Poppins?
If people lost the ability to recognize sarcasm and satire?
Such a population would be stupid beyond comprehension. How would they understand complex information underlying health care, much less become health care providers? It would be a world of chiropractors, acupuncturists, and naturopaths.
At least I am glad I do not live in such a society and can rest easy knowing that the future for my children is in good hands and will not have to sell their children to the rich for food.

How accurately do physicians estimate risk and benefit?
A new study suggests that physicians tend to overestimate the benefits of treatments, tests, and screening tests, while also underestimating harms.
Scott Gavura on January 26, 2017
Link


The era of medical paternalism has largely disappeared. Few of us are willing to permit health decisions to be made on our behalf, without our input. Today the goal is “shared decision making,” which describes a collaborative process that takes into account a health professional’s medical knowledge and advice, but also a patient’s own preferences and wishes. Truly shared decision-making includes an explicit consideration of the evidence base and a consideration of a treatment’s (or test’s) expected benefits and potential harms. Decisions may vary, as patient values can vary. There isn’t always one “right” answer. 
A friend recently asked me for my opinions of statins. This fellow is in his early fifties, is active, and doesn’t smoke. His physician had brought it up, but didn’t push him in one direction or another. As this particular type of decision is so common, there are an abundance of decision tools out there that use the evidence from studies of cardiovascular risk, allow you to enter your own risk factors, and then generate a summary of the expected benefits and harms of a statin (like this one.) Using the tool, and a discussion with his physician, he decided against taking a statin. The potential for benefit was still small, and he didn’t want the daily hassle of medication, the potential side effects, and the prescription cost. This was the right decision – for him, and it was a great example of shared decision-making that was guided by a tool that incorporates the best clinical evidence. Unfortunately, too few of our health care decisions are accompanied by direct clinical evidence and effective decision aids.
In order to make health decisions that reflect our own preferences, we need to understand the benefits and risks of any test, treatment or screening test that is offered to us. The foundation underlying these decisions is an honest, unbiased evaluation of the scientific evidence. Science is the best tool we have to objectively evaluate new treatments, and understand their risks and benefits. Back in 2015 I wrote about a systematic review that assessed patient expectation of benefits and harms of treatments, tests, and screening tests. It looked at 36 studies: 16 that looked at treatments, and 20 that looked at screening. The sample was small but the findings were consistent: patients tend to underestimate the harms and overestimate benefits of a medical decision. But patient decisions can be heavily influenced by expert opinion. So do experts have accurate understandings of risk and benefit? That’s what this new study sought to determine.
This new paper is entitled ” Clinicians’ Expectations of the Benefits and Harms of Treatments, Screening, and Tests.” Published in JAMA Internal Medicine, it’s written by Tammy C Hoffmann and Chris Del Mar of the Centre for Research in Evidence-Based Practice at Bond University in Australia. They were the authors of the systematic review of patient expectations, so this is a nice companion document. As context, this is a systematic review, which is a review and analysis of previously published research. By methodically selecting and analyzing trials, the intent of a systematic review is to give a perspective on the totality of the evidence base on a particular subject. Ultimately systematic reviews are only as good as the evidence they include. As we’ve seen with alternative medicine, systematic reviews can lead to some questionable findings.
This was a systematic review of all studies that assessed physician expectations of the benefits and/or harms of any treatment, test, or screening test. Any study in which participants provided a quantitative estimate was included. The authors did a comprehensive search of four databases and then, using responsive articles, attempted to identify additional articles (via references and index words). There were two authors, so most of the work appears to have been done in parallel. If studies provided an answer about benefits or harms, the authors calculated the proportion of participants that responded correctly or over/underestimated the answer.
The authors found a total of 48 eligible studies published between 1981 and 2015, encompassing over 13,000 clinicians. Twenty studies examined treatments, 8 addressed tests or screening, and 20 addressed medical imaging. Across the studies, 181 outcomes were assessed, 23% assessing benefits and 69% assessing harm expectations (8% were neutral). Overall, physicians correctly estimated 11% of the benefit expectations and 13% of the harm expectations. Most participants overestimated benefit and underestimated harm.
Expectations of benefit
Benefit expectation were assessed in 15 studies, which included studies such as risk reductions from statins, or reduction in breast cancer mortality due to mammography. The findings are a bit complicated to sort out as the primary data is reported in different ways, which sometimes limited a calculation of over/underestimation. In studies with overestimation/underestimation data, 50% or more of the participants overestimate benefit for 7 outcomes (32%) and underestimated benefit for 2 outcomes (9%).
Expectations of harm
Estimates of harm were compared with correct estimates in 26 studies (69 outcomes). Most participants (>50%) correctly estimated harm for only 9 outcomes, with most underestimating harm for 20 outcomes (34%) and overestimating harm for 3 outcomes (5%).
The paper goes on to break out the findings by intervention type, which I won’t describe. Instead, here’s Figure 1 which shows the overall summary of benefits (Figure 2, showing harms, is here).

Is there a “therapeutic illusion”?
This is the most comprehensive summary to date of physician expectations. Clinicians rarely estimated benefits and harms accurately, and tended to underestimate harms and overestimate benefits. Over-optimism has the potential to push physicians to offer, and patients to accept, more interventions than might be necessary or desirable. And that may be what we’re seeing. Certainly initiatives like Choosing Wisely are expressly designed to guide patients and clinicians into re-evaluating low-value or ineffective care.
The reasons for these beliefs are unclear, and given the diversity of studies included, there’s almost certainly many factors that contribute to an evidence gap. It could reflect the sheer volume of clinical evidence that exists, or it could be challenges in translating clinical trial results to individual assessments of risk and benefit. It’s not clear, and while the authors of this paper summarize the possibilities, ultimately, “more research is required.”
So recognizing that clinicians are likely overestimating benefit and underestimating harm, how do we fix the problem? Shared decision-making is going to be biased towards testing and treatment if both patient and clinician have unfounded optimism. Decision support tools, when grounded in good evidence, like the one I described above, are a start. More generally, clinicians need access to unbiased, accurate evidence summaries that describe benefits and harms, and help them translate this to patients accurately. Ultimately if we want to improve the efficiency and effectiveness of care, we need to focus on both health professional and patient expectations. Without addressing it, our health professionals and our health systems will continue to offer sub-optimal, and even wasteful or harmful, medicine.

Less benefit, more risk. Our assumptions about health treatments are probably wrong.
Scott Gavura on April 23, 2015
Link


I’m a health professional, but sometimes a patient as well. And like most patients, I generally don’t want health decisions being made without my input. Yes, I want the best medical information, and the advice of medical professionals, but ultimately I want to make my own decisions about my care. That’s the norm in health care today, but relatively new in the history of medicine.
Medical paternalism, where patient preferences are secondary (or even ignored), is disappearing. Even informed consent, where patients are given information on risks and benefits, doesn’t adequately describe the drive towards a two-way exchange, with an empowered, engaged patient. Today the goal is shared decision making, which describes a mutual decision that is informed by a health professional’s medical knowledge and advice, but also incorporates a patient’s own preferences and wishes. Truly shared decision-making includes an explicit consideration of a treatment’s expected benefits and potential harms, yet reflects patient values.
Screening is a textbook example of why shared decision-making should be our goal. Given the benefits of a disease screening program may be modest, and not without harms, understanding and incorporating individual preference is essential. Some may value the small but incremental benefits of screening, and choose to be screened despite the risks of false positives, investigations, and possible overtreatment. Given the exact same circumstances, another individual may opt to forgo screening, making a different, yet equally acceptable decision. While there are some health interventions for which the benefits are unequivocal, and others for which the harms are just as clear, most health treatments (and interventions like screening) have both benefits and potential harms that must be carefully assessed within the context of patient preferences. Research published earlier this year has identified a significant barrier to truly effective shared decision-making and risk assessment: Across a wide range of interventions, we routinely overestimate the benefits of health treatments, and underestimate their risks. 
There’s extensive research to show that patients want to be engaged and informed in health-care decision-making. Critical to this approach is an honest consideration of the scientific evidence. One of the important benefits of this blog has been to highlight the risks of poor health decisions when treatment considerations fail to consider plausibility, and instead rely on other forms of evidence. High-quality scientific evidence is the best tool we have to evaluate new treatments and determine their risks and benefits. However, that evidence must be informed by a consideration of basic scientific principles, and the overall scientific evidence established on a topic. For example, there is no persuasive evidence that alternative medicine treatments like acupuncture or homeopathy give any objective medical benefits. Yes, you can find individual clinical trials that may be statistically significant. But the totality of the evidence, when considered in the context of what we know to be a fact in fields like physics, biochemistry and other disciplines, makes it far more likely that positive effects seen in any single trial are due to bias, poor design, or statistical noise. Homeopathy and acupuncture have “effects” that, when closely scrutinized, are indistinguishable from placebo. Yet it’s unlikely that a potential recipient will have this fact disclosed to them by a provider. And without this information, consumers really can’t make a truly informed decision – and shared decision-making is effectively impossible.
It’s not just alternative medicine where effective shared decision-making can be difficult. Health professionals can also be at fault. As a pharmacy student I was taught to help patients make better self-care decisions, such as how to choose an appropriate cough and cold medication. I have probably assessed and recommended products to hundreds of patients over the years. I admit to facing some considerable cognitive dissonance later in my career, when I took a much closer and skeptical look at the science and determined that there’s little evidence to support the use of many of the ingredients in cough and cold products. Moreover, these products were best avoided in children, where the limited evidence that did exist suggested that the harms probably outweighed any expected benefits.
Opiates are another example where health professionals have created expectations of benefit without balancing the risks of harms. Over-prescribing climbed for years, based on inappropriate assumptions of benefit and risk. Today we are faced with a significant public health issue, with tremendous harms to patients.
If health professionals, whether they be pharmacists, physicians or surgeons, are overly optimistic about the treatments they are recommending, then their recommendations may contribute to overtreatment, or in the case of screening, overdiagnosis. And that can also compromise the goal of effective shared decision-making.

A patient’s own expectations can change treatment decisions, too. If patients believe a treatment or health intervention is beneficial, they are more likely to seek it out. This is well documented, from those that seek antibiotics for viral infections, or those that “ask their doctor” about the latest prescription drug (or homeopathic nostrum) that’s widely advertised and may not be as effective (or as safe) as advertised. And as has been well documented at this blog, the infamous Dr. Oz effect that creates an overnight demand for the latest “miracle” is regrettably not based in anything that resembles good science or medicine.
What appears to be the first paper to systematically examine patient assumptions and expectations has now been published. It lays out a significant challenge to optimal health-care decision-making.
The study, by Tammy C. Hoffmann and Chris Del Mar, was published earlier this year in JAMA Internal Medicine. It is a systematic review of other studies that have assessed patient expectations of benefits and/or harms of any treatment, test or screening test. They cast a very wide net in searching for evidence. They included studies where participants were asked to estimate the expected benefits/harms of a treatment, test, or screen. There had to be some form of quantification of the benefit in the questions asked.
For each identified study, the authors calculated the proportion that estimated accurately, and those that over/underestimated. The results were striking and consistent.
Of over 15,000 studies assessed, only 36 papers were eligible for inclusion. Study dates ranged from 1994 through 2013. Sixteen studies looked at treatments, and 20 looked at tests or screening.
Expectations of benefit

Of the 32 studies that assessed expectations about benefits, the authors compared the study outcomes with the “correct” estimates of benefit. The majority of participants overestimated benefits for treatments and interventions like cancer screening, the efficacy of infliximab for inflammatory bowel disease, and for cancer surgery. Only two studies found accurate estimates of risk, in studies of the expected benefits of cataract surgery and cervical test smear accuracy. In only one trial did participants underestimate benefit: surgery for low back pain (see figure 2, above).
Expectations of harm

In the 11 studies that examined expectations of harms, at least 50% of participants underestimated harms for 10 of 15 outcomes studied. These harms included the negative consequences of cancer screening, the side effects of different medications, and complications from surgery (see figure 3, above). Correct estimates of harm were noted for only two interventions: glasses after cataract surgery, and miscarriage risk from amniocentesis. Harm was overestimated for only one outcome, that of breast cancer risk due to hormone therapy.
Despite the diversity of studies included in this review, the overall findings are persuasive. In general, patients tend to overestimate benefits and underestimate the harms of treatments. This is an important signal to advocates of science-based medicine, suggesting that our continued advocacy in areas like the regulation of homeopathy are important. But beyond getting the science right, we need to work to find better strategies and tools to inform decision-making. Decision aids have been identified as one tool that may support both shared decision-making and an accurate assessment of risks and benefits. The Choosing Wisely campaign is another approach, designed to support dialogue between patients and providers with the goal of minimizing the use of low-value treatments and interventions.
Conclusion
Health care providers and patients alike want the best care. Treatment decisions need to be informed by accurate, science-based information. Health professionals need to not only be aware of the benefits and harms of the treatments that they recommend, they need to disclose them in an accurate and candid way. They also need to recognize that their own biases, positive or negative, may be inflating patient expectations about benefits and risks. There’s much more that can and must be done to improve how we collaboratively make health decisions.
Reference
Hoffmann TC, Del Mar C. “Patients’ Expectations of the Benefits and Harms of Treatments, Screening, and Tests: A Systematic Review .” JAMA Intern Med. 2015;175(2):274-286. doi:10.1001/jamainternmed.2014.6016. PMID 25531451.

Nature vs. Technology
Nathanael Johnson has written an interesting book about what “natural” means, and doesn’t mean, drawing upon his own life but more importantly – the scientific literature.
Harriet Hall on March 18, 2014
Link
For those who dismiss advocates of the “natural” as ignorant of science and deluded by the logical fallacy that natural = best, Nathanael Johnson’s new book is an eye-opener: All Natural: A Skeptic’s Quest to Discover if the Natural Approach to Diet, Childbirth, Healing, and the Environment Really Keeps Us Healthier and Happier. If nothing else, it is a testament to the ability of the human mind to overcome childhood indoctrination in a belief system, to think independently, and to embrace science and reason.
Nathanael Johnson was brought up by hippie parents who subscribed to every “natural” belief and fad. His mother nearly died of a postpartum hemorrhage when he was born at home (he weighed 11 pounds!). His parents didn’t report his birth, and he didn’t have a birth certificate. He co-slept with his parents, never wore diapers (imagine the clean-up!), was allowed to play in the dirt and chew on the snails he found there, was fed a Paleolithic diet, was never allowed any form of sugar, didn’t know there was such a thing as an Oreo cookie, was home-schooled, and did not know that public nudity was taboo until he and his brother shocked the folks at a church picnic by stripping naked to go swimming in the lake. Nudity was customary in his home, and he was encouraged to “let his balls breathe.”
As he grew up, he started to question some of the dogmas he had learned from his parents. He had been taught that good health resulted from forming connections with nature, but he found that nature “generally wanted to eat me.” Now an adult and a journalist, he understands science and how to do research. He tried to read the scientific literature with an unbiased mindset, asking questions about the subjects in his book’s title rather than looking for evidence to support any prior beliefs, and he arrived at pretty much the same conclusions we science-based medicine folks did. But he still appreciates that a natural approach has value, and he seeks to reconcile nature with technology. He calls his book a comfortable refuge from people who are driven to extremes.
When his wife was pregnant, he interviewed home birth advocate Ina May Gaskin, visited birthing centers, looked up statistics about C-sections and fetal monitoring, and decided what he wanted was “No Nonsense Evidence-Based Midwifery.” He and his wife found exactly that in a hospital where nurse-midwives delivered the babies and high-tech care was immediately available for emergencies. All went well, but throughout his wife’s labor and delivery he worried about whether they had made the right choice. Afterwards, he realized that more worry was in store: no matter how he tried to protect his newborn daughter, she would, in the course of her life, be hurt, would suffer, and would eventually die. A nurse told him “Whenever there’s uncertainty or discomfort, people tend to want to fix it. We have absolutely no tools in this culture for simply accepting, but that’s what you have to do sometimes.” These are wise words that patients might do well to consider before seeking the false certainties offered by alternative medicine.
He investigates food and identifies three faulty assumptions:
     1.  Molecules Matter, Food is Irrelevant. He says he stopped reading labels because the same type of food molecules may have different consequences if they arrive in a slice of coconut, a steak, or a scoop of gelato.
     2.  Everyone Is the Same. No, some of us can digest lactose and others can’t; and we are difference in many other ways. Evolution has molded humans to eat diets as diverse as humanity itself.
     3.  Institutions, Not Individuals, Should Be in Charge of Diet. We learn food preferences and food traditions through our culture. Adjusting nutrients at a national level reduced the risks of goiter and neural tube defects and eradicated vitamin deficiency diseases like pellagra, beriberi, and rickets, but obesity became a problem. “In attacking the nutrient-efficiency problems we created a super-sufficiency problem.”
He decides that eating scientifically is impossible because no diet advice is well-supported by science. Both science-based diets and natural diets go beyond the evidence and produce gurus who claim certainty where there is complexity. They are like two sides of the same coin. He argues for rediscovering the pleasure of food rather than just eating to satisfy hunger or to eat the “right” number of calories of the “right” foods. At the same time, he realizes that telling people to eat what they enjoy would be catastrophic for those with metabolic disorders and laughable for those who can’t afford it.
He describes his brief and disastrous experimentation with a raw food/live food diet. He discovers that when he thought he was avoiding toxins, he was just ingesting different toxins, plant toxins. He says yes, there is probably something out there trying to make us sick or eat our brains, but there is very little certainty about which toxins are harmful to the human body in what doses over a lifetime. It would be a worthy goal for science to identify all toxins and remove them from our diet, but “I can’t wait that long for dinner.”
He investigates vaccines. He offers the insight that when parents ask a pediatrician about vaccine risks, they are looking for advice from someone who has seriously considered the risks and the objections of anti-vaxxers rather than just dismissing them. Simple reassurance that science has decreed vaccines necessary is counterproductive. He counters the argument that we have insufficient proof that vaccines are safe by citing Wendell Berry’s advice that the trick is not to find certainty, but to act thoughtfully with partial knowledge. (That’s the trick of science-based medical practice, too.) He comes to a new understanding of vaccines as a “natural” means of protecting children from disease. He realizes that “unnatural” means things he doesn’t understand. The more he learned about things like vaccines, the more natural they seemed.
He puts a new spin on the old canard that doctors only treat symptoms, not underlying causes. After his appendectomy, he asks the surgeon why it happened to him and why it happened when it did. He gets only a vague answer. “Conventional medicine is concerned with helping pragmatically, using the information available to accomplish what it can…you don’t have to know why a fire started to put it out.”
He talks about placebos, describing Benedetti’s research. He suggests there is a role for metaphor in medicine. His mother tried “Earthing” and felt better, less driven, less scattered, better able to enjoy the moment. She realized it might be a placebo effect, but she felt a sense of connection with the earth. He suggests that she was being “healed by a metaphor.”
He argues for a kinder, gentler practice of medicine. He thinks the Flexner report had the effect of eclipsing the patient. And certainly, part of the appeal of “integrative medicine” is the way it returns the focus back to the “whole” individual patient. In a way, it is “a manifestation of love.” He tells about a doctor who prescribed suicide drugs for terminal patients and found that none of his patients actually used them. Their discomfort was not caused by their disease, but by their loss of control. He wished he had realized that was the underlying problem and had been able to remedy it more effectively. “When healing is reduced to a battle between technology and disease, patients lose both responsibility and control.”
He tells about a pediatrician who spent hours educating his patients about asthma, diabetes and allergies, and measured his success by the fact that for the last 5 years of his career not a single patient from his practice had to go to the ER for complications of those diseases. He voluntarily cut his own paycheck by $50,000 a year by teaching the parents of hemophiliac children to administer clotting factors at home instead of coming to his office each time.
Johnson investigates organic farming, industrial pig farms vs. small family operated pig-friendly farms, and forestry management for multi-use purposes rather than as a single crop. He even investigates raw milk, recognizing the dangers of infection, but also learning that milk from grass-fed, pampered cows has a different microbial flora that might conceivably have a beneficial effect on health by protecting humans from more harmful bacteria. He suggests there may be ways to achieve the economies and successes of mass food and lumber technology with more animal-friendly, tree-friendly, and human-friendly methods. He makes it sound as if it’s definitely worth a try.
We are all isolated beings struggling for survival, but we are also part of a larger whole. The technological perspective builds a protective barrier around us; the natural approach sees us as part of a nurturing whole, invites nature in, and fosters relationships. Nathanael Johnson argues for neither the technological nor the natural perspective, but for reunification. He says:
I can’t do without technology: I’m not willing to give up antibiotics, or movies, or ice cubes, or germ theory, or space exploration. But I’m also dismayed by the way faith in technological progress tends to trade away beauty, and wonder, and joy, and all of those slippery, unquantifiable things that – in the end – make life worth living.
There’s a lot of good science and common sense in this book. It gave me a better understanding of what those “natural” advocates are thinking, and of ways in which modern medical practice could be improved. The only quibble I had with it is that he talks about right brain/left brain differences that are not substantiated by recent evidence.

True believers, entrepreneurs, and scammers in alternative medicine
In the online echo chamber promoting alternative medicine, there are varying degrees of deception. There are true believers (who are often victims), entrepreneurs (who are often true believers who found a profitable business), and scammers. The categories are not mutually exclusive.
David Gorski on February 19, 2018
Link

Long ago, when there was no Internet outside of the military and some colleges, where the average person had no access and even if they did there was no world wide web, dinosaurs roamed the earth, and I remember encountering quackery. Back then, unlike now, it came mainly in the form of books, magazines, newsletters, or quacks giving talks, and it is true that all of those things still exist. However, they have been relegated very much to secondary (or even tertiary) status by the existence of the Internet, the web, and, above all else, social media. Back when I started my hobby of writing about medicine and quackery, blogs and websites were the main media from which quackery flowed. Now, these, too, tend to be secondary, with Facebook, podcasts, and especially YouTube becoming the major places in social media where quackery is promoted.
In fact, video seems to rule these days, and no self-respecting quack would be without a YouTube channel, complete with videos that he could also post on his Facebook page. The more “complete” quacks even have documentaries around which they build their brand, an excellent example of which is the antivaccine propaganda film masquerading as a documentary entitled VAXXED: From Cover-up to Catastrophe. Not only is there the movie, but there’s a popular antivaccine Facebook page and YouTube channel to which the film’s producers Del Bigtree and Polly Tommey, antivaccine activists both, regularly post videos, which range from them ranting about vaccines to them showing interviews with parents who mistakenly believe that their children were “vaccine-injured.” Occasionally, even Andrew Wakefield himself deigns to show up!
Oddly enough, I got the idea for this post from a place that I normally wouldn’t have expected, an article on Babe.net, “A YouTuber who claimed being vegan cured her cancer has died from cancer,” which reminded me of another article I saw on the very same source, “This vlogger says you can cure cancer with happy thoughts and juice — and she’s making money doing it.” Basically, the reason is that these posts, particularly the latter one, show how not only do YouTube and social media provide the means for quacks to spread misinformation in a far more effective and targeted manner than was ever possible before, but how lucrative they can be. That led me to—of course—think about the man who is perhaps the ultimate example of how to monetize quackery through social media, Mike Adams, given that his NaturalNews.com empire was the subject of an article last week by Jonathan Jarry of the McGill Office for Science and Society. These “alternate realities” harm medicine and vulnerable patients every bit as much as the “alternate realities.”
I like to divide these sorts of promoters of quackery into three categories: True believers, entrepreneurs, and scammers. There is, of course, considerable overlap, and it’s possible for one source to be all three at the same time, but usually one category predominates
Mari Lopez: A true believer killed by alternative medicine
Let’s start with one of the most common kinds of alternative reality social medial presences, that of patients with cancer who provide “testimonials” of how they “cured” their cancers. I’ve written about such testimonials more times than I can remember going back to the very beginnings of this blog. In most cases it’s pretty easy to see the mistake in attribution that led the patient to believe that whatever quackery he was using cured him (I’m talking to you, Chris Wark!), but sometimes you can’t really know for sure until after the person giving the testimonial dies of her cancer, as in the case of Kim Tinkham.
Let’s take a look at her story, because even as she promoted alternative medicine for cancer she became its victim:
A YouTuber who said that eating a raw vegan diet, drinking juice and praying to God cured her cancer has passed away.
Mari Lopez, one half of the vlogging duo Liz & Mari, succumbed to an aggressive terminal cancer that spread to her blood, liver and lungs.
She was known for posting viral videos about how she was able to cure her stage four diagnosis through faith in God and following a strict diet of raw vegetables.
Liz Johnson, it turns out, is Mari Lopez’s niece. Here are the videos that started it all, first part 1:
And then part 2:
It doesn’t take long into the first video to discover one of Mari Lopez’s other claims:
Mari referred to how she was able to cure her cancer in 90 days by drinking juice. She said her renewed faith and diet even pushed her to renounce her lesbian sexuality.
“I was healed by God and faith and used to live a gay lifestyle,” she explained in one video.
So, according to Mari and Liz, not only did her vegan diet, prayer, and juicer cure her of cancer, but it “cured” her of homosexuality as well. In this lengthy video, we learn a lot more about her “journey”:
First, we learn in her interview with her niece Liz Johnson that thirteen years before, Mari Lopez had stage II breast cancer at age 37, which means either that it was large and without positive lymph nodes or that the tumor had spread to three or fewer lymph nodes under the arm. She underwent a mastectomy and chemotherapy and radiation. Basically, her treatment sounded appropriate for her disease, and I suspected that her cancer was estrogen receptor-positive because estrogen receptor negative cancers, if they recur, usually don’t recur so late; estrogen-responsive tumors do, sometimes even 20 years later. In any event, she stopped taking the pill early. Later thirteen years after her original treatment, Lopez’s cancer recurred in her lungs and liver and “everywhere,” as she put it in the video.
Now, in retrospect, I can explain why she did so well for so long, given that the first videos are from the summer of 2016. Lopez’s tumor was relatively indolent (as many estrogen receptor-positive tumors are, even after they metastasize), and she describes being offered an anti-estrogen treatment after her metastatic diagnosis, which she took for 30 days and then stopped because she “didn’t feel it” in her spirit. She could easily have done fairly well for a couple of years without any effective treatment before she started feeling sick last summer as her tumor progressed.
Particularly telling is her answer to this question: “Can you prove your healing with medical records? Are you completely cancer free?” You’ll note that she says she’s cancer-free, but studiously avoids the question about whether she could prove it with medical records, saying that it’s “over and done with” and that she “feels it in her spirit” that she’s “healed.” She even goes on to say, “I really don’t care if nobody believes me,” continuing, “Will I go to a doctor? Yeah, if I have to, if I feel the need, if God leads me to going in that direction, I have no problem going and getting checked.”
To this, Liz Johnson replies, “A lot of the evidence is in just seeing how you are now.”
It turns out that the diet that Mari Lopez was promoting was typical “alkaline diet” quackery, only not from Robert O. Young, but from a quack I hadn’t heard of before, Dr. Sebi (real name Alfredo Bowman, which should tell you something right there that this herbalist adopted a pseudonym to sell his products), who represents his diet as an “African approach” to curing cancer that he dubbed Dr. Sebi’s Cell Food. It sounds similar to Robert O. Young’s “pH Miracle Living” program, but that’s because all these diets are very similar: Vegan, lots of vegetable juices, lots of supplements ± coffee enemas. There is no good evidence that such diets impact the course of cancer. Lopez’s use of them was the same as not treating her cancer, only she spent a lot of money and effort.
In the Babe.net story, we also see something very disturbing but all too common in alternative cancer cure testimonials where the patient ends up dying:
Speaking to babe from Houston, Texas, Liz explained her aunt was pushed by her sister (Liz’s mom) to start radiotherapy and chemotherapy, which she thinks disrupted her diet of juicing.
Liz refers to this as “inconsistencies,” which she thinks pushed Mari to her death. Liz’s mom favored conventional medicine while Mari wanted to follow alternative treatment.
“She was following a raw vegan style [diet],” Liz explained. “My family is not familiar with that style of living… What happened was, as Mari was living with my mom, my mom started to tell her that she needed to eat meat now. She said it was OK to use things that she didn’t want to use. My aunt was very against the microwave because of cancer-causing issues with that, and my mom would cook her things using the microwave.”
I can’t recall how many times I’ve heard variants of this sort of blaming of the victim or someone urging the victim to undergo conventional therapy. The beginning of the end of the narrative often begins when the believer can no longer deny that she is getting sicker, that her cancer is progressing, and decides, either because of family pressure or on her own, to undergo conventional therapy near the end. Inevitably, the believers will claim that the chemotherapy or the radiation or whatever conventional therapy the cancer victim decided to undertake is what kept the quackery from working. Basically, they blame the victim for not being faithful enough. In this case, Johnson is despicably blaming Lopez’s mother for pushing the evil microwaved food on her and blaming Lopez’s sister (Johnson’s mother) to undergo chemotherapy. Yes, she blames her mother and grandmother for her beloved aunt’s death! I wonder how that goes over at family gatherings.
In his post about this case, Steve Novella makes the point that this aunt-niece team conflated faith healing, in which Lopez believed that God healed her, with the alternative cancer treatment Lopez chose, saying:
Both, in fact, are manifestations of faith. Alternative medicine is often a mixture of pseudoscience and faith healing, blended seamlessly together.
The faith claims have a significant problem with logic and consistency. I don’t begrudge anyone their personal faith, but I do think it is counterproductive to depend for anything important on a mythical being you just hope exists. What I always find intriguing is those who claim that God cured them of a disease, but not really. In this case Liz believes that God cured Mari of her cancer, but Mari wavered, so the cancer came back and killed her. Really?
Noting:
This is another absolutely typical part of the faith healing narrative, borrowed heavily by the alternative medicine narrative. Failures are always blamed on the patient. It is a convenient rationalization – they lacked sufficient faith in the case of the former, and they did not adhere fanatically enough to the regimen, for the latter.
Precisely. Worse, the faith that drove Mari Lopez and Liz Johnson to such absolute certainty makes their videos that much more convincing to non-physicians, many of whom wouldn’t even notice how Lopez tap danced around the question of whether she’s ever been demonstrated by imaging studies to be cancer-free. They might not even care about how ultimately the duo started to monetize their faith by offering premium Vimeo clips. So confident does Liz Johnson remain in the faith healing and “alkaline diet” that she vows to leave all the YouTube videos up and make the premium Vimeo videos free. She even did this against her aunt’s wishes, as near the end of her life her aunt asked her to take the videos down:
Liz explained that towards the end of her life, Mari had asked her to take down the videos proclaiming the healing powers of the vegan diet. Liz said she would keep the videos up, genuinely believing they could give cancer patients help. She even called the experience a “test of faith.”
“She didn’t want anybody listening to her, which I understood because I knew that she was depressed,” Liz explained. “She was upset because her cancer had come back. I can understand how maybe she didn’t want to give people false hope. I had to pray a lot about it, because I could feel her pain. But in my own relationship with God, I knew it could still help people, it could still make a difference in people’s lives, it could still give hope.”
As Mark Crislip would say, I got nothing. So I’m going to move on to an entrepreneur, Brittany Auerbach.
Brittany Auerbach, a.k.a. MontrealHealthyGirl: Quack entrepreneur
One thing I’ll say about Harry Shukman’s Babe.net articles, he doesn’t pull any punches. In this one where he discusses Brittany Auerbach’s vlog, he leaves nothing to the imagination:
Few diseases are scarier than cancer, as anyone who’s had it will tell you – nerve-racking days in waiting rooms, sleepless nights weighing up the odds of your death. It’s only human that at your most vulnerable, you’ll go online to research any advice on how to handle your diagnosis.
Which is why you’d have to be a real piece of shit like Brittany Auerbach to try and profit off of cancer patients looking for a shred of hope. Auerbach is a prominent health vlogger with over a hundred thousand followers on her YouTube channel, MontrealHealthyGirl. Posting videos looking like your typical internet personality, she rakes in millions of views from desperate people by peddling pure, utter bullshit.
I also notice on her Instagram account that she’s a naturopath. She describes herself thusly on her blog:
Hi! My name is Brittany and I am a certified Naturopath and passionate Health Coach who gets the honor of assisting people all over the world in reclaiming their health naturally!
A 3-year battle with Interstitial Cystitis, IBS, severe acne, thyroid disease and cardiac arrhythmia… and the eventual healing of my body has led me to begin this blog! My health struggles inspired me to pursue a career in the alternative health field as well as to create several e-books such as Fully healed, fully guilt-free, The smoothie Challenge, The candida cleanser and Fully Nourished in the hopes of sharing accessible, convenient and affordable healthy alternatives and natural disease-fighting information with the world!
I note that she sells a three-month program of “personalized treatment” for whatever ails you that she will not begin until payment in full of $395 is received via PayPal. Her book describing her three-day Candida juice cleanse costs $55 (as do all of her books) and a package of her e-books (of which there are only four books, by the way) costs $197, a whopping discount of $23 off the price of purchasing them all separately!
“Not-a-Doctor” Auerbach also lays on the quackery heavily in her YouTube videos, “Heal Your Cancer Overnight: You absolutely can reverse this!”
In it, she claims that if you are still alive, it’s not too late no matter how far along your cancer is and that you can definitely extend your life or even reverse your cancer with her methods. Here, she advertises her “Best Stage 4 cancer reversal protocol: Do this immediately to heal terminal cancer quickly!”
And here she tells you how to heal all viruses, including HIV, herpes, Epstein-Barr, and Ebola, “naturally”:
Early in the video, we learn that she’s an antivaxer too, as she says that if you’re dealing with viral issues you should stay far away from vaccinations, noting the amount of “heavy metals” in vaccines, the “toxins,” and the additional “viral load” put “directly in the bloodstream” (yes, she uses that old antivaccine trope) is serious and will make your health worse.
As for cancer, she appears to be pushing a variant of the same old “alkaline diet” cancer quackery:
Overnight you can stop new cancer cells from growing. It’s important that people realize how quickly this really happens. Once you start flooding your body with alkaline minerals, enough water, the right nutrition, loads of rest, peaceful, loving thoughts. Surround yourself with people that love you. Do activities that you enjoy. Feel hopeful and excited. Live the life that you’ve wanted to live your whole life and haven’t. Let go of those negative emotions. Do what you can. All of those tools can be adopted right now, and guess what, they will change your internal chemistry overnight.
That’s right. You can reverse cancer just by thinking happy thoughts. Truly, she is repeating the central dogma of alternative medicine, in which wishing and thinking make it so, all the while rattling off easily refuted alternative medicine canards about chemotherapy and cancer-causing foods. She is also full of the arrogance of ignorance; she is the Dunning-Kruger effect personified:
I don’t tell you these things for fun. I don’t tell you these things because I’m guessing. I tell you these things because I know. I’m super educated about the workings of the body. And I really understand chemistry, and everything that I do, every single healing modality that I use, is based on rebalancing body chemistry to trigger inner healing, faster cell birth, and changing the state of your body so that disease cannot continue…I’m a chemist. I can physically balance your chemistry.
It turns out that Auerbach was, at the time the story was being put together, at the Hippocrates Health Institute in Florida, which has been a frequent topic here because its owner Brian Clement preyed on First Nations girls with cancer in Canada with his phony cancer cures and because HHI represents an utter failure of regulating health care.
Clearly, Brittany Auerbach is a true believer, but she is further along on the believer to scammer scale. She believes she’s healed herself of a variety of chronic diseases, but she’s also become a naturopath and now makes her living selling the same quackery she believes in. However, to see the next stage in evolution, we need to consider the case of Mike Adams.
Mike Adams: In a category all his own
I assume that nearly all regular readers here know who Mike Adams (a.k.a. The Health Ranger) is, namely the man responsible for one of the largest, if not the largest, repositories of quackery on the entire web, NaturalNews.com Unfortunately, he’s managed to poison my Google reputation rather handily with at least three dozen posts over the last couple of years claiming that I worked with cancer chemotherapy fraudster Dr. Farid Fata (I did not and never met the guy, who was never on the faculty of my cancer center), that he reported me to the FBI and Michigan Attorney General (if he has, two years later I have yet to hear from either of them), and even tried to insinuate that I was a pedophile, while leaving plausible deniability.
Adams has a long and ignominious history online, having started out (as best as I can tell) selling Y2K scams. From the beginning, Adams was talented. He saw the possibilities in web marketing to drive traffic to his sites and use that to monetize them very early on. In this, we can see him honing his early techniques. Indeed, he took it far beyond just that, mastering the dark arts of using “black hat” search engine optimization, running link farms, and using those skills to drive traffic back to his site. It turns out that the skill set that made Adams so talented at crafting mass e-mail marketing campaigns that actually persuaded the marks to give up their money is the same skill set that he later honed to become an expert at SEO. He also has a history of selling software to be used for spamming, back when email spam was a huge concern, and used to boast about his software’s ability to evade antispam software. And, of course, before Google cracked down on the practice for purposes of increasing a website’s Google ranking, Adams used to run a huge server farm, with hundreds of different domain names (Internet sites) providing thousands of web links back to Natural News. He’s very big in the world of online alternative medicine and, in the era of Donald Trump, has aligned himself with Trump and the alt right. He’s even been punished more recently for trying to game Google to his advantage, with NaturalNews.com being briefly delisted from Google, which led predictably to his usual overblown hysterics.
Even now, though, when such crude methods of increasing one’s Google juice have largely been neutralized by search engine software coders, Adams still has his ways. I was reminded of this last week by an article by Jonathan Jarry published on the McGill Office for Science and Society entitled “Mike Adams Is Building an Alternate Reality Online.” What Adams is doing now is basically an extension of what he’s been doing all along, all to make money:
Much has been written recently about online “echo chambers”: the idea that we are catered to on the Internet with sites and recommendations that reinforce our preexisting beliefs. If you watch a lot of science videos on YouTube, follow many scientists on Twitter, and regularly search for scientific questions on Google, your online experience will shift away from neutrality, as search results, post sorting, and recommendations will be tailored to your pro-science stance. This is an echo chamber because, in due time, you only hear your beliefs repeated back at you and stop seeing what’s happening on the other side.
Echo chambers for the pseudoscience crowd exist as well, though Mike Adams’ online bubble is so vast and self-sufficient, it warrants the term “ecosystem”.
The ecosystem consists of:
A bit of online sleuthing revealed that Mike Adams owns over 50 websites. The topics they cover go beyond alternative medicine and help shape an entire worldview: fear of medicine and science (gmo.news, medicine.news, vaccines.news), anti-Left and pro-freedom hype (campusinsanity.com, libtards.news, freedom.news), and doomsday prep advice (survival.news, collapse.news).
But Adams’ ecosystem goes beyond just a bunch of websites that all extensively link to one another. In fact, he explicitly designs tools to try to keep you in his ecosystem:
To help bring you back within the confines of this online ecosystem, Mike Adams offers you a toolbar for web browsers so that, regardless of which website you visit, NaturalNews.com is never more than a click away. If you were thinking of searching Wikipedia for a particular food or nutrient, stop. Mike Adams owns NaturalPedia, which offers biased health information such as his page on naturopathy. On it, you will “learn” that it can treat irritable bowel syndrome and ulcers. At the bottom of the page, you will see ads for iodine, magnesium, and colloidal silver because, like the denouement you predict halfway through watching a bad movie, you will have inferred by now that Adams has a store.
Jarry forgot TruthWiki, which is also owned by Adams. (Actually, no he didn’t. He just did here; it’s mentioned later in the article.) He also has a search engine:
Mike Adams has a search engine. If you’re looking for a replacement to Google that “filters out corporate propaganda and government disinformation”, Adams suggests you use Good Gopher. Searching for “Washington Post”, for example, I was not shown the actual website of the Washington Post; instead, I was sent to TruthWiki, RealInvestigations.News, Disinfo.news, and a whole alternate reality in which the newspaper churns out fake news and is beholden to Monsanto. Some of the “independent news websites” that Good Gopher sticks to and that aren’t owned by Mike Adams include far-right website Breitbart.com and Alex Jones’ conspiracy epicentre Infowars.com.
Of course, Mike Adams used to have a show on Infowars.com. Why he and Alex Jones parted ways is unclear, although I’ve always suspected that it might have had something to do with Jones’ decision to become a competitor selling his own line of supplements.
Adams even has his own social network. Jarry is puzzled by one aspect of Adams’ online empire, namely his PubMed search engine, and it does sound puzzling initially. Jarry quite rightly suspected some sort of biased search algorithm, but a friend of his couldn’t find any clear evidence of bias. Personally, my guess is that Adams is probably just monitoring what topics people are interested in and where the IP addresses come from, something he wouldn’t get otherwise. There might be other reasons as well. For instance, if you look at his search engine, there are icons representing various topics, such as “Harmful Toxins,” “Heavy Metals,” “Healing Therapies,” “Harmful Therapies,” “Nutritional Supplements,” “Superfoods,” and more. If you click on any of them, a list comes up, complete with links to PubMed searches on them, and to the right a list of related articles from one of Adams’ websites conveniently appears. I suspect that’s the reason right there, to try to keep people in his ecosystem.
Jarry even provides a nice infographic listing many of Mike Adams sites in his online empire of disinformation:
Near the end, Jarry also notes:
Raising awareness of this “alternate reality” online is important since the world of alternative medicine can seem, to the casual observer, quite benign. Behind the curtain of empathy and so-called holistic care, however, often lies a darker notion: that modern medicine cannot be trusted. Few alternative medicine proponents reach the near-operatic heights of Mike Adams, but his empire of misinformation has major ramifications. In the age of the digital echo chamber, his voice can be heard even if you don’t go searching for it.
And it’s true. Adams’ empire is so ubiquitous now that it’s hard to avoid it even if you are trying to avoid it. Inevitably someone you know will share a link from it with you on Facebook. You’ll see links posted on Twitter and other social networks. You’ll see articles from Natural News shared by e-mail. They’re everywhere.
Believer, entrepreneur, or scammer
As I said at the beginning, I believe that in the world of online alternative medicine promotion, there is a spectrum that goes from the believer to the scammer. None of the three categories are mutually exclusive and it is possible—common, even—for one person to have elements of one or more. For example, clearly Brittany Auerbach is a true believer, but she’s risen to the level of entrepreneur because of how much she shills for her own products. Her online persona is designed mainly to sell her product. In contrast, Mari Lopez and Liz Johnson were clearly true believers, but their primary purpose was not to sell product. They did sell some subscriptions to their premium Vimeo videos, but little else. Their main motivation was to spread their message of faith and quackery.
Adams is a trickier case. He’s clearly gone beyond the level of entrepreneur, having been selling things online, including many dubious things, at least since the late 1990s. I’ve always tended to lean towards his being more purely a scammer, but it’s hard to say. There could be an element of true belief there. However, his behavior, his tendency to attack his enemies, the sheer magnitude and complexity of his business model, and his attempts to rope people into his ecosystem and keep them there, the better to sell them alt right paranoia, supplements, disaster prepper gear, and conspiracy theories, makes me lean towards characterizing him as a scammer. This opinion is bolstered by his risible attempts to slap a scientific veneer on his fear mongering and quackery.
Another case worth considering is that of Belle Gibson, who was also in the news again recently. She was a darling of the wellness movement, a lifestyle philosophy promoted by largely white, largely female social media mavens, a prime example of which is Gwyneth Paltrow’s Goop. Lots of New Age mysticism is usually involved, along with trendy diets and exercise plus a heaping helping of quackery. Gibson had a lucrative business going in this industry based on her claim to have cured herself of cancer without doctors. Ultimately, investigative journalists started looking into her claims, including claims of donating money from her products to various charities (which she did not), and her whole story started falling apart. Basically, Gibson was revealed to have been lying about having had cancer.
Therein is the dark side of the wellness industry that makes it ripe for scammers like Gibson and Adams:
…I think what’s happened in the last two years is that Belle Gibson has become the face of the dark side of the wellness movement. The wellness movement has been around for decades.
It was born out of the New Age revolution, and it was always meant as an adjunct to conventional medicine. It wasn’t meant as a replacement of conventional medicine.
What’s happened in more recent years with insurance companies and chiropractic clinics and big business co-opting the word, is that that meaning is being lost.
Then on Instagram people have almost medicalised the term so they’re using wellness as a way of treating different ailments. Then to go further than that, which is what Belle Gibson did, they then promote this distrust in conventional medicine as well.
Combine people like Mike Adams (and even Brittany Auerbach) and the power of social media, and this dark side will continue to do harm.

Mainstream Doctors Actually Do What CAM Claims They Don’t
Alternative medicine proponents criticize mainstream medicine and think they can do better. Evidence from medical journals shows that their criticisms are not valid.
Harriet Hall on October 24, 2017
Link

Advocates of complementary and alternative medicine (CAM), including naturopaths and functional medicine practitioners, constantly criticize mainstream medicine. The recurrent themes are:
    • Doctors don’t do prevention
    • They only prescribe drugs and surgery and don’t consider non-drug treatments
    • They don’t treat the underlying causes of disease
    • They don’t treat the whole person
    • They don’t know anything about nutrition
    • When they criticize CAM, they’re only trying to protect their turf and increase their income
And of course, CAM advocates present themselves as prioritizing prevention, using non-drug treatments, finding the underlying cause, treating the whole person, being experts in nutrition, and apparently implying that they are not financially motivated (!).
I’ve been catching up on my CME (Continuing Medical Education) and I currently have four recent biweekly issues of American Family Physician on my desk – the flagship journal of the American Academy of Family Physicians. Family physicians rely on it to keep up to date; there is a quiz in each issue that allows members to earn CME credits. It arguably represents the best in current clinical practice, the standards primary care physicians should aspire to (even though some may not measure up in practice). I thought it would be interesting to review those four recent issues looking for evidence for or against the accusations of CAM advocates. I found plenty of evidence that those accusations are wrong.
August 15th
    • Preventing Falls in Older Persons (a comprehensive review addressing the whole patient and recommending many non-drug approaches – see details below)
    • Putting Prevention into Practice (discourages routine pelvic exams, which would tend to reduce physician income)
    • Health Literacy Tools in the Outpatient Setting (a sensitive approach to making sure patients are able to understand medical information)
    • Accepting Family Care in Chronic Illness (a patient’s perspective, intended to improve physician empathy and understanding of patients’ nonmedical needs)
    • Pregabalin Does Not Decrease the Pain of Sciatica (a recommendation against using the drug Lyrica)
    • TOP POEMS [Patient-Oriented Evidence that Matters] of 2016 Consistent with the Principles of the Choosing Wisely Campaign (including numerous recommendations to avoid using pharmaceuticals or surgery for specific conditions)
September 1st
    • Aseptic and Bacterial Meningitis: Evaluation, Treatment and Prevention (yes, prevention, with vaccines and chemoprophylaxis for close contacts, preventive measures that have been proven effective)
    • Vitamin D for the Management of Asthma (reports a Cochrane review evaluating a “natural treatment”)
    • Noninvasive Treatments for Low Back Pain (including exercise, spinal manipulation, heat, massage, tai chi, yoga, and advice)
    • Varenicline for Smoking Cessation (a drug treatment, but an effective way to help patients prevent smoking-related morbidity and mortality; no non-drug treatment is as effective)
    • Combination of Resistance and Aerobic Exercise Best for Older Persons with Obesity (combination of exercise and weight loss better at improving function than either alone; specific advice about what kind of exercise is best)
September 15th
    • Diabetes Self-Management: Facilitating Lifestyle Change (a comprehensive review of lifestyle changes and how to achieve patient compliance – see details below)
    • Vitamin B12 Deficiency: Recognition and Management (recognizing a nutritional deficiency and treating it appropriately)
    • Screening for Syphilis Infection in Nonpregnant Adults and Adolescents, in the Putting Prevention into Practice series (early detection can prevent late-stage disease and transmission to others)
    • Low Back Pain: American College of Physicians Practice Guideline on Noninvasive Treatments (recommends 13 interventions, from exercise to yoga, that should be tried before prescribing medications)
October 1st
    • Recurrent Ischemic Stroke: Strategies for Prevention (lifestyle modifications like exercise, smoking cessation, and decreased alcohol are covered before discussing anti-clotting drugs)
    • Secondary Hypertension: Discovering the Underlying Cause
    • Pet Therapy: Helping Patients Cope
    • Yoga for Cancer-Related Symptoms in Women with Breast Cancer (recommended for depression and other symptoms)
Preventing falls in older persons
This comprehensive article covers prevention, addresses the whole person, and recommends non-drug interventions that have been proven effective. It recommends screening older patients for a history of falls or balance impairment, looking for underlying causes (everything from cancer to inappropriate footwear), evaluating home safety, exercise to improve strength and balance, review of medications, vision correction, foot care, and a vitamin (D). They recommend multifactorial interventions that address every aspect of the whole person.
How does this differ from the approach of a naturopath? It doesn’t, except that naturopaths add a few things like silica, strontium, Curcumin-Boswellia blend, or other supplements that were not included in the American Family Physician article because they are not supported by good evidence. I know of no evidence that naturopaths are any better at preventing falls than MDs.
In a YouTube interview, a naturopath fails to mention most of the things in the American Family Physician and focuses on diet. She recommends assessing for malnutrition and correcting or preventing sarcopenia by increasing protein intake and taking leucine. She recommends evaluating gut health to ensure nutrients are absorbed, and checking for low magnesium levels. I know of no evidence that that approach reduces the incidence of falls.
Diabetes: Facilitating lifestyle change
Mainstream medicine agrees with alternative medicine that lifestyle changes (weight loss, exercise, healthy diet, etc.) are indicated to treat (and help prevent) diabetes and pre-diabetes. And of course, these measures are good advice for anyone wanting to stay healthy and reduce the risk of chronic diseases. But it has been notoriously difficult to get patients to make those lifestyle changes. The article in American Family Physician provides all kinds of techniques for improving compliance, techniques that are equally applicable to achieving compliance with any treatment for any health conditions. They recommend inquiring into patient expectations, attitudes, and preferences, and addressing psychosocial factors (i.e., treating the whole person). They address effective timing of interventions, assessing a patient’s readiness to change, setting clear goals, individualizing the plan, encouragement, empowerment, dealing with slips and relapses, teaching coping mechanisms, controlling cues, providing social support, etc. There is a wealth of wisdom here that any health care provider could benefit from.
What passes for prevention in the world of CAM
CAM claims to stress prevention, but they are arguably not as good at prevention as mainstream providers. Patients of CAM providers are less likely to get vaccinated or to get evidence-based screenings like mammograms.
Some chiropractors recommend routine maintenance adjustments and claim that keeping the spine properly aligned will prevent most or all illnesses. This view is not supported by evidence.
Acupuncturists claim to provide preventive care. There is no evidence that regular acupuncture treatments prevent anything.
Homeopaths also claim to offer prevention; and as a substitute for effective vaccines, they offer homeopathic remedies that contain no active ingredient.
Some people believe that if you just eat right, you won’t ever get sick. That’s demonstrably not true, and recommendations about how to eat right vary widely.
Mainstream medicine invented prevention. It developed proven preventive measures like the vaccines that have saved untold lives. CAM has contributed nothing of substance.
Conclusion: CAM is unnecessary
It is clear from the examples above that mainstream medicine does not do all the bad things CAM criticizes it for. And that it does do all the good things CAM claims to do.
CAM’s criticisms of mainstream medicine are misguided. Good mainstream clinicians do at least as good a job as CAM providers when it comes to prevention, using science-based non-drug treatments, treating the (real) underlying causes of disease, treating the whole person, understanding the principles of nutrition, and achieving good health outcomes for their patients. All these things fall squarely in the province of good science-based medicine. Science-based medicine offers a wide range of proven treatments, while CAM offers treatments that have not been proven to work (if they had, they would be “medicine,” not “alternative medicine.”)
When mainstream doctors criticize CAM, it is not to protect their turf. They routinely follow the evidence and stop using treatments or tests that are lucrative when they are found to be ineffective. Their goal is not to maximize income, but to do what is best for their patients. They want patients to be able to give informed consent, to have accurate information about treatments that are ineffective, untested or inadequately tested, scientifically implausible, or hyped way beyond any all-too-meager evidence.
CAM has tried to appropriate things like prevention and exercise from mainstream medicine. They talk as if they discovered those things. They didn’t. They borrowed freely from mainstream medicine without acknowledgement, and they haven’t added anything of significant value.
Mainstream medicine is far from perfect. One big problem is the limited time doctors have with patients, and CAM has capitalized on that flaw; they often spend more time with patients, but they don’t make as good use of that time as doctors could. It makes sense to concentrate on improving mainstream medicine. It makes no sense to replace it with even more imperfect substitutes. We don’t need CAM.

Science, Evidence and Guidelines
Mark Crislip on June 15, 2012
Link
Disclaimer: I am a paid Medscape blogger and writer, and since they are in part supported by advertisements from the Pharmaceutical companies, indirectly I am in the thrall of Big Pharma.
I found Harriet’s post on the Medscape Connect topic of How do you feel about Evidence-Based Medicine? interesting.
I wondered about the breakdown of the comments by both specialty and opinions about SBM. So I read the 226 comments and classified them by field and response. I classified each response as disapprove, approve or nuanced. It is not, obviously, a legitimate survey and there was more than a little subjective interpretation in deciding how to classify the responses. I have no doubt that others would get different results; it is not methodologically sound analysis. The discussion was in the Family Medicine & Primary Care section, so it is unlikely to be representative of any population, including that of Family Practitioners and Primary Care Physicians. I would bet, as in alternative medicine and most topics, Shruggies predominate and are the silent majority.
Even though I belong to what a commentator referred to as the not so silent “militant wing” of SBM, I was surprised at my results:
Medscape Results
Speciality	Disapprove	Approve	Nuanced
Family Practice	19	2	6
General Practice	4	1	1
Psych	13	1	3
Neuro	11	1	1
Anesthesia	9	1	1
Prevention Medicine	1		
Radiology	2		1
ID	1		
ER	3	2	1
Internal Medicine	9	2	1
Ortho	5		
Dermatology	2		
Occupation Medicine			2
Surgery	7	2	
Peds	3	1	1
Allergy	1		1
Ob	6		1
Endocrine	2		
Pulmonary	3		
ENT	2	1	
Ophthalmology	1		
Other	4	1	4
Urology	4	1	1
Oncology	2		1
Cardiology	1	1	1
Critical Care	3	1	1
Pulmonary	3		
GI	1	1	1
Pathology	2		
Total	124 (72%)	19 (11%)	29 (17%)
It was the lack of nuance that amazed me. That docs had issues with EBM/SBM should be given. I have issues with SBM, but like democracy, it is superior to all the alternatives. And the inability to differentiate between EBM/SBM and the guidelines upon which they are based flabbered my gaster. I have never been able to come to grips with the concept that people who should know better, don’t appear to. Being perfect myself, I am taken aback when others are not.
There is an intellectual hierarchy in medicine, and at the top, of course, are infectious disease doctors. I leave it the commentators to rank the various specialities based on their response to EBM, since my referral base may be reading this and work is slow enough as is thanks to all the quality initiatives to decrease hospital acquired infections.
Besides, any list would be based entirely on confirmation bias and the personality of the specialists I work with. I also work in an internist dominated, inpatient medicine teaching hospital, so my bias is towards evidence and science, since I have be able to quote chapter and verse when I teach the residents. What makes me expert in Infectious Diseases is partly experience, which has made me a better diagnostician every year I am in practice, and my futile attempt to master the ID literature, which makes me better at choosing a treatment. But does the greatly extolled (at least for the EBM naysayers) experience make me better at treatment? I don’t know for sure, but I doubt it
Some examples. P. acnes is an anaerobic bacteria that causes occasional post-operative back infections and artificial joint infections. It is an odd bug, being such a part of the normal flora it can fester for years with little in the way of classic inflammatory signs and symptoms. It is resistant to metronidazole, my usual go to drug for anaerobes. The treatment is usually high dose penicillin, and I see maybe a case a year, tops. I have a smattering of cases where penicillin did not appear to be effective, but when changed to clindamycin the patient rapidly improved. Perhaps they would have improved anyway; most do. But as a result I have this nagging unsupported by the literature ‘feeling’ that clindamycin is better for P. acnes.
Or MRSA. From the published data, all the current treatment options, when compared to a beta lactam for susceptible organisms, suggests that all antibiotics for MRSA stink on ice. From the amount of data, i.e. preponderance of information, the rank order, and rank it is, for treating MRSA is perhaps vancomycin = daptomycin = linezolid => quinolone or TMP/Sulfa plus rifampin > ceftaroline (for MRSA pneumonia it is linezolid > vancomycin > ceftaroline = quinolone or TMP/Sulfa + rifampin with no daptomycin on the list). What I think, in my Eldunarí, that the real order is ceftaroline > daptomycin > linezolid = quinolone or TMP/Sulfa + rifampin with vancomycin last, with lots of caveats depending on the specific staph and the organ infected. I have no doubt that other ID docs would rank the drugs differently. My experience, bias, and interpretation of the literature can be conflicted depending on how I approach what I am reading and the patients I have treated in the past; although I bet the literature will catch up with me someday. It always does 🙂
It is an interesting challenge when treating a given individual since the study population may not be reflected in my particular patient and then applying bias and hubris becomes very tempting. I am very careful when I talk to house staff to differentiate my recommendations that are based on studies, based on extrapolation from known data and those that are my opinion, based in part on the three most dangerous words in medicine (in. my. experience).
It requires remarkable intellectual effort to not give in to my bias and not come up with rationalizations to justify my opinion. Working in a teaching hospital makes that easier, since I have to make sure the residents know the ‘right’ answers so they will pass their boards. And someday they will be in practice, having to make decisions without the support of the faculty, and have to make the ‘right’ decisions that will benefit their patients as well. Right being in quotes as a recognition that medicine changes, and I get to bore/amuse the residents about what we did in my day, when every PVC received lidocaine, asthma was treated with aminophylline, we had no CT scans and I wore an onion on belt, as was the style of the day.
I have long noticed that many docs do not think like me (i.e. an internist) and I was influenced years ago by an article in the Annals (sorry Harriet) called Reflections on Internal Medicine and Family Medicine by an FP doctor. Obviously, training for breadth will have a different emphasis than training for depth. A metropolitan subspecialist needs a vastly different skill set than a small town generalist and that will be reflected in their training and approach to medicine
The money quotes
Internists are trained in the classic deductive reasoning model of differential diagnosis. Family physicians are trained in this model of problem solving on internal medicine services, but, in practice, they often take a less structured and more empiric approach to clinical reasoning that is based in clinical epidemiology.
and
Internists, especially those in academic medicine, may underestimate the degree of nonconformity and rebellion required of U.S. medical students who entered family medicine in the 1970s and 1980s. It is these students who now fill the faculty ranks of family practice programs around the country. Nowhere is this nonconformity more evident than in the ambivalence with which family physicians have integrated themselves into academic medicine. The importance of this counterculture mentality in the character of the family physician can best be understood by reading the works of Gayle Stephens, which are considered classics in family medicine. Stephens is one of the philosophical fathers of the family practice movement, and his book, The Intellectual Basis of Family Practice, has inspired a generation of family physicians. An understanding of the culture of family practice will help to explain much of the independence and cynicism that so often seem to characterize the family physician.
So I should not surpassed the overwhelming antipathy revealed in the Medscape comments by those in the field of FP and GP, which were particularly dismissive of evidence, often erroneously confusing evidence with guidelines.
In my own field the guidelines have remarkably little evidence to back them up, although in ID we do have the distinct advantage in treating individual patients that it is often the purpose to eradicate large swaths of microbial life; I am glad there is no Karma or I would dread the payback in the next life. Outcomes in ID are often more binary than most specialties; either I cured your infection or I did not.
Still the guidelines in ID, as noted in the ID literature, are lousy with opinion instead of evidence:
Approximately one-half of the recommendations in the current guidelines are supported by level III evidence (derived from expert opinion). Evidence from observational studies (level II) supports 31% of recommendations , whereas evidence based on >1 randomized clinical trial (level I) constitutes 16% of the recommendations… The IDSA guideline recommendations are primarily based on low-quality evidence derived from nonrandomized studies or expert opinion.
They say that the IQ of a committee is the average IQ of the committee divided by the number on the committee and it occasionally shows in the guidelines. Sometimes the advice is stupid, or at least brain dead. The pneumonia guidelines suggest, with no data, “In general, when switching to oral antibiotics, either the same agent as the intravenous antibiotic or the same drug class should be used.” That simple comment is a sure way to jack up costs and breed resistance. Not that my opinion is necessarily better, although my IQ is divided by one, but most patients with uncomplicated pneumonia who respond promptly probably require no more than amoxicillin or doxycycline, although I can pepper that statement with several pages of caveats. It still amazes me that for a common infection like pneumonia, we still do not really know the optimal treatment:
Most patients with CAP have been treated for 7–10 days or longer, but few well-controlled studies have evaluated the optimal duration of therapy for patients with CAP, managed in or out of the hospital. Available data on short-course treatment do not suggest any difference in outcome with appropriate therapy in either inpatients or outpatients.
Or take the acute bacteria rhinosinusitis (ABRS) guidelines
High-dose” (2 g orally twice daily or 90 mg/kg/day orally twice daily) amoxicillin-clavulanate is recommended for children and adults with ABRS from geographic regions with high endemic rates (≥10%) of invasive penicillin-nonsusceptible (PNS) S. pneumoniae.
High dose amoxicillin high dose should be enough to treat penicillin-nonsusceptible S. pneumoniae, at least for a relatively trivial infection like sinusitis. But here is the deal: S. pneumoniae doesn’t make a beta lactamase, an enzyme that degrades amoxicillin, which they point out in the text. For S. pneumoniae, clavulanate, which inhibits the beta-lactamase, adds nothing to the treatment over amoxicillin except diarrhea and cost and is recommended for the odd H. influenza in the sinus of adults, which does make a beta lactamase, as they mention:
H. influenzae was isolated in 36% of patients with positive bacterial cultures consistent with ABRS, compared with 38% for S. pneumoniae and 16% for M. catarrhalis. Unfortunately, the rate of β-lactamase–producing H. influenzae was not reported in this study.
After their convoluted argument, they finally conclude
Thus, the recommendation of amoxicillin-clavulanate in adult patients with ABRS is primarily based on in vitro susceptibility data and the current prevalence rates of β-lactamase production among H. influenzae.
Why even bother? Not confidence inspiring, although I understand the reasoning, it is awkward at best: You want to give more amoxicillin if you have high endemic rates of invasive penicillin-nonsusceptible (PNS) S. pneumoniae but due to the potential for H influenza you want amoxicillin-clavulanate instead. I suspect the commentators on Medscape who complain about guidelines were not doing so due to a fine reading of the content in the context of an understanding of an extensive literature, but more in rebellion against being told what to do by the man: insurance companies and the smarting pants academics. I share their concerns, but for different reasons.
There is science and evidence, and often I do not really know what do with the results, even when impressive. Recently the NEJM published an article that demonstrated that azithromycin, when compared to other agents, was associated with ‘excess deaths’ The overall increase in deaths was small, although not if you were one those in the excess.
“Relative to amoxicillin, azithromycin was associated with an increased risk of cardiovascular death (hazard ratio, 2.49; 95% CI, 1.38 to 4.50; P=0.002) and death from any cause (hazard ratio, 2.02; 95% CI, 1.24 to 3.30; P=0.005), with an estimated 47 additional cardiovascular deaths per 1 million courses; patients in the highest decile of risk for cardiovascular disease had an estimated 245 additional cardiovascular deaths per 1 million courses.”
The paper is EBM/SBM, an epidemiologic study. How do I approach such a study? First, I will admit, is a gut check. Was I surprised at the headline? Nope. Quinolones, azoles and macrolides are all known to have effects on the cardiac conduction system. This is in contrast with a report that quinolones were associated with an increase in retinal detachment. While quinolones are known to affect connective tissue, I am a bit more skeptical about this one. I would like to see it confirmed.
The azithromycin study I am more likely to take to heart. The problem with the study is that while it demonstrated those with cardiac problems were more likely to have the adverse reaction,the main indication for the antibiotics was sinusitis or bronchitis. Will the same effects happen for other infections? What is the risk and benefit?
For example, in both sepsis and pneumonia, receiving a macrolide is associated with increased survival compared to those who do not. Also, acute pneumonia, where there is survival benefit from a macrolide, is one of many infections with increased vascular events, including arrhythmias and heart attacks. Since macrolides are part of the guidelines for the treatment pneumonias, are we seeing more deaths from the drug or the infections for which it is being used? That being said, I would be less likely to use azithromycin for ‘trivial’ infections like sinusitis and bronchitis that I would for more life threatening infections.
What do you do if you think experience trumps data and evidence? The risk was increased from 47 to 245 per million doses azithromycin given, making it unlikely that an experienced provider, with 3 to 6000 patients will ever be likely to inadvertently kill a patient with azithromycin. In my experience, I never killed a patient with azithromycin. That you know of.
That is the problem with experience based medicine. Experience will always be insufficient to recognize rare yet important complications or the beneficial effects of therapies applied to populations such as treating hypertension. SBM should be the basis of medicine, but its application will often require the ‘art’ of medicine.

Rationalizing the Ridiculous: Acupuncture
Mark Crislip on September 19, 2014
Link

I remain flummoxed. How do physicians and health care systems, trained in all the sciences that lie at the heart of medicine, justify the use of pseudo-medical interventions with no basis in reality? Rationalization. Making excuses:
a defense mechanism in which controversial behaviors or feelings are justified and explained in a seemingly rational or logical manner to avoid the true explanation, and are made consciously tolerable – or even admirable and superior – by plausible means.
Rationalization of the ridiculous comes in many forms. It has been said that it is a mark of a first rate intelligence to able to hold two contradictory ideas in the mind at the same time and still retain the ability to function. Clever as it is, I suspect the opposite is true.
Acupuncture: Ignoring your own analysis
So often rationalization is as simple as denying your own data where the content of a pseudo-medicine paper is at odds with its conclusions. The most egregious example is the Cassidy study on stroke and chiropractic. But other examples are easy to find. Take “Acupuncture and related techniques in ambulatory anesthesia“, a review of the use of acupunctures (regular, electric and acupressure) for postoperative pain and nausea.
They start off with an abstract that says:
Research suggests that acustimulation may alleviate postoperative morbidities, although the body of evidence of the effect is equivocal.
‘Suggests’, ‘may’, ‘equivocal’. Hardly a ringing endorsement. The introduction is equally lukewarm:
trials of hands-on-therapies face inherent difficulties in the design of placebo controls, and the body of evidence is equivocal. The use of sham acustimulation as a placebo control is inclined to have high placebo effects
They compared a hodgepodge of acupunctures, with variable outcomes, some positive and some negative. There were very few double blind studies and most with a positive effect for a subjective endpoint (pain or nausea) often had no change in objective endpoints (use of pain or nausea medications).
If this were a real intervention based on real physiology, I would conclude the intervention did nothing. As an intervention with no prior plausibility, the results are even less impressive.
In this case? Well, they say:
With reservations, as some of the trials reviewed are small and must be perceived with caution, the results suggest that acustimulation may be beneficial in alleviating postanesthetic morbidities. The techniques are easy to perform, and adverse events and costs are minimal. In particular, acupressure is noninvasive, well tolerated, and may have a good patient acceptability. Acupressure can be applied by medical staff, patients or parents and maintained after discharge. A feasibility study shows neither delay in the surgical procedure, nor additional anesthesia time attributed to the introduction of acupuncture and acupressure. It may be profitable to consider implementing acustimulation to ambulatory patients.
I presume they mean profitable in the sense of beneficial or useful.
It is really remarkable how a therapy that in this analysis is questionable at best (and in other analyses worthless) is still suggested because it is cheap and easy to apply, not because it is effective and based on sound medical evidence. It is difficult to imagine such a low standard for approval of an antibiotic. But with antibiotics you would expect a real effect against a real infection.
Acupuncture clinics
The National Research Center in Complementary and Alternative Medicine, Institute of Community Medicine in Norway, might suggest using acupuncture for postoperative pain and nausea. In the US major medical institutions do not suggest, they act. For example:
Loyola announces opening of new acupuncture clinic for patients with pelvic health disorders.
The usual nonsense is put forth to justify charging patients for fantasy-based therapies:
These conditions place a significant burden on our health-care system and the economy due to the prevalence and the fact that many patients don’t respond to traditional treatments, said Larissa Bresler, MD, a medical acupuncture doctor, Loyola University Health System, and an assistant professor of Urology, Loyola University Chicago Stritch School of Medicine. The goal of the clinic will be to offer patients another tool to help alleviate their symptoms without the side effects of medication.
That is the ‘since air travel and airplanes have problems we should rely on flying carpets’ argument.
Acupuncture has been used as a healing tool for more than 2,000 years.
The ever-popular appeal to antiquity logical fallacy, and not even a true appeal since the use of needles is a recent development and as practiced at the turn of the century TCM was gruesome:
Chinese doctors own that they know nothing at all of surgery. They cannot tie an artery, amputate a finger or perform the simplest operation. The only mode of treatment in vogue which might be called surgical is acupuncture, practised for all kinds of ailments. The needles are of nine forms, and are frequently used red-hot, and occasionally left in the body for days. Having no practical knowledge of anatomy, the practitioners often pass needles into large blood vessels and important organs, and immediate death has sometimes resulted. A little child was carried to the dispensary presenting a pitiable spectacle. The doctor had told the parents that there was an excess of fire in its body, to let out which he must use cold needles, so he had pierced the abdomen deeply in several places. The poor little sufferer died shortly afterwards. For cholera the needling is in the arms. For some children’s diseases, especially convulsions, the needles are inserted under the nails. For eye diseases they are often driven into the back between the shoulders to a depth of several inches. Patients have come to us with large surfaces on their backs sloughing by reason of excessive treatment of this kind with instruments none too clean.
The Loyola press release mentions the mechanism by which acupuncture is alleged to work:
Acupuncture is based on the traditional Chinese teaching that energy flows through the body along channels called meridians. Illness occurs when energy is blocked or disrupted. The insertion of needles into designated points improves the flow of energy and releases feel-good hormones, beneficial chemicals and immune system cells to reduce inflammation, aid healing and promote pain relief.
Without mentioning that meridians and the energy it contains are nonsense, divorced from physiology and anatomy and has never been demonstrated to exist.
They will use acupuncture for
…pelvic pain, nausea from pelvic surgery, interstitial cystitis, painful bladder syndrome, postoperative pain, prostatitis and overactive bladder.
The reviews for use of acupuncture for the above conditions are typical – maybe it works but the studies are poorly done:
Evidence for the efficacy of needle acupuncture for treating the disorders evaluated remains inconclusive. The intervention showed promising results for reducing pelvic and back pain during pregnancy and climacteric vasomotor symptoms, although well-designed studies are needed to make the results more precise and reliable.
And:
The evidence that acupuncture is effective for chronic prostatitis/chronic pelvic pain syndrome is encouraging but, because of several caveats, not conclusive. Therefore, more rigorous studies seem warranted.
Which is the same as it ever was. Acupuncture only has effects in studies with poor methodology. The better the study, the worse the effect until well-designed studies that remove all potential biases fail to show acupuncture has any effect.
Which, given the prior plausibly that acupuncture would be effective for anything is zero, is what would be expected. But always more studies need to be done.
It hardly seems a solid foundation for a clinic for treating and charging the ill. I always wonder why the Board of Trustees of these institutions allows the use of magic in their hospitals.
But I hope that, since the acupuncturist involved is also a urologist, there will finally be an answer to the question that has puzzled me for years: why they are no meridians and acupoints in the male genitals? Why no life energy flowing in the life-generating organs? You can probably guess my answer.
Integrating naturopathy
Forget acupuncture, what about using naturopaths? “Integrating naturopathy: can we move forward?” The short answer is no. I have discussed both naturopathic education and ND fitness as primary care providers in the past.
The gist of the article is the author wants to dump all his difficult-to-treat patients on naturopaths:
With so many of our patients presenting with chronic pain, functional disorders, and other difficult-to-manage chronic conditions, allopathic primary care physicians clearly need more compelling options for managing chronic disease…Naturopaths would seem ideally positioned to fill this partnership role.
Clever in a Machiavellian kind of way. Play up the holistic BS and send them off to the mostly harmless, although mostly fantasy-based, therapies that make up naturopathic practice. If you have a troublesome patient what could be better than to identify another practitioner who can take care of them better than you? As a resident, transferring a patient to another service (from medicine to surgery was most common) was not only for the patient’s benefit, you had a transient dip in the work load and no longer had a patient whose issues (bad gallbladder) you were not comfortable managing.
The only problem is that referring to naturopathy as a credible source for medical diagnosis and treatment doesn’t make it so. To say:
Naturopathic medicine is a distinct method of primary health care—an art, science, philosophy and practice of diagnosis, treatment, and prevention of illness. In stark contrast to a narrowly interpreted biomedical model, where pharmaceuticals are administered to battle disease, Naturopathic physicians seek to restore and maintain optimum health in their patients by emphasizing nature’s inherent self-healing process …. This is accomplished through education and the rational use of natural therapeutics. The naturopathic physician thus uses holistic approaches to enhance and restore the body’s own innate healing systems. Naturopaths undergo a 4-year graduate-level course of study but are generally not residency trained.
Ignores the fact that they are trained in unproven, often irrational, fantasy: nutritional supplements, homeopathy, acupuncture, hydrotherapy and electrical stimulation. There is no nonsensical therapy that is not in the naturopathic armamentarium. That is part of the problem with naturopathy: any and all medical magic is in their purview. And they are usually anti-vaccine.
The author recognizes there are conflicting paradigms between doctors and ND’s. You think? He gives several examples of the different approaches between MD and ND for some common diseases, but suggests that somehow these differences can be overcome. It would is like recommending raising an alligator as a household pet. In the end, it will end badly. Say, honey, have you seen the dog and the baby?
Despite his suggestions that education and medical standards could somehow integrate ND’s into a reality-based practice of medicine, to do so would require the ND to abandon the fantasies in which they have invested time and money. I doubt that will happen.
Establishing and improving lines of open, respectful, constructive communication will be a first step toward developing the type of collaboration between allopathic and naturopathic physicians that our patients deserve.
Our patients do not deserve naturopathic practice, although the lateral pass of the difficult patient might make clinic easier. To quote me, if you mix cow pie with apple pie, it does not make the cow pie taste better; it makes the apple pie worse. Our patients need a better apple pie, not cow pie.

Caution vs Alarmism
Steven Novella on July 1, 2015
Link


When I lecture about the need for science-based medicine (SBM), I have to pause about half-way through my list of all the things wrong with the current practice of medical science, and I balance my discussion by emphasizing what I am not saying: I am not saying that medical science is completely broken. It is just really challenging, we need to raise the threshold for what we consider reliable higher than most people think, and there are some practical fixes we can do, some of which are already in the works.
It is easy, however, to “demonize” any person, institution, or philosophy by taking all the negative aspects that are inevitably present and wrapping them up in a frightening package, perhaps throwing in some conspiracy thinking or sensational alarmism.
Take, for example, a recent article by F. William Engdahl, “Shocking Report from Medical Insiders“. The headline alone warns you that you may be in for some sensationalism.
For background, Engdahl is a journalist with some ideas that are out of the mainstream. He believes the theory that oil is not biological in origin but geological, and therefore “peak oil” is a myth, specifically designed to create panic in the public for political control. He also believes that the warnings about global warming are exaggerated. He is the author of Seeds of Destruction, which in my opinion is a propaganda hit piece against the technology of genetic modification.
In other words, Engdahl appears to be a journalist with an ideology, with the result being some skillfully written but ultimately misleading pieces.
His recent article is based upon recent statements made by medical editors about the state of medical research. He quotes Dr. Richard Horton, Editor-in-chief of The Lancet:
Much of the scientific literature, perhaps half, may simply be untrue. Afflicted by studies with small sample sizes, tiny effects, invalid exploratory analyses, and flagrant conflicts of interest, together with an obsession for pursuing fashionable trends of dubious importance, science has taken a turn towards darkness.
Actually I think 60% may be a better figure than half. This should not be news to regular readers of SBM. We often highlight the problems of poor methodology: overreliance on P-values and P-hacking, researcher bias, publication bias, testing implausible hypotheses, failure to replicate, citation bias, and conflicts of interest.
We don’t deny these very serious problems exist within the institution of medical science. Our point in exploring them, however, is two-fold:
     1.  To provide a helpful guide to distinguishing reliable science from unreliable science. There is still good science out there, and with it we can come to confident conclusions.
     2.  We want to improve the practice of science by fixing the problems. You can either point to these known issues as if they are part of some dark conspiracy, or you can point to them in a constructive way in order to advocate for change.
Horton, to his credit, is doing the latter. He goes on to write:
Instead of changing incentives, perhaps one could remove incentives altogether. Or insist on replicability statements in grant applications and research papers. Or emphasise collaboration, not competition. Or insist on preregistration of protocols. Or reward better pre and post publication peer review. Or improve research training and mentorship. Or implement the recommendations from our Series on increasing research value, published last year. One of the most convincing proposals came from outside the biomedical community. Tony Weidberg is a Professor of Particle Physics at Oxford. Following several high-profile errors, the particle physics community now invests great effort into intensive checking and rechecking of data prior to publication.
That list should also be familiar to regular readers of SBM. Engdahl doesn’t quote that part of the paper or even mention its existence, because it does not serve his conspiracy alarmist narrative.
Engdahl also quotes Dr. Marcia Angell, previous editor-in-chief of the New England Journal of Medicine (which he misnames the “New England Medical Journal“).
It is simply no longer possible to believe much of the clinical research that is published, or to rely on the judgment of trusted physicians or authoritative medical guidelines. I take no pleasure in this conclusion, which I reached slowly and reluctantly over my two decades as an editor of the New England Journal of Medicine.
Angell, of course, is referring in part to the many problems with published medical science as discussed above, and I agree with much of what she says. However, in my opinion she overstates the situation as well. I shared a stage with her about ten years ago at an alternative medicine conference. She seemed far more interested in taking on the pharmaceutical industry than pseudoscience itself, and said as much.
There is much to criticize in how the pharmaceutical industry has tried to subvert medical research to their advantage. This is why they need to be carefully monitored and regulated. They are not the only ones, however. I would argue that the entire supplement and CAM industries are far, far worse, and they are not as well regulated.
The problems in science can be fixed
There are many endemic problems with the science of medicine as it is currently practiced. It is possible to draw the wrong lessons from these problems, often influenced by one’s ideology.
While a great deal of published research is preliminary, implausible, or of low methodological rigor, not all of it is. We can still pick through the research to find the most reliable studies that have survived independent replication. I agree that, with any lower threshold, you might as well flip a coin.
In addition to looking at the literature with open and critical eyes, a valuable lesson is that we need to make serious efforts to fix identifiable problems with the institutions of science (as outlined above).
The big challenge, as Horton himself points out, is not that we don’t know what to do, but that no one has the power to do it. There is no central power hierarchy of science, and when power is distributed it is often difficult to accomplish major changes. Everyone thinks it is someone else’s responsibility.
What we need is for the issue to become well known, to cross that nebulous threshold of political will. We need many voices pointing out the problems that need fixing, and we need to put pressure on those with any power.
The problem with conspiracy-mongering is that it is often not constructive — there is no call to action, just the sense that the world is hopelessly corrupt. Engdahl concludes his article with:
Corruption of the medical industry worldwide is a huge issue, perhaps more dangerous than the threat of all wars combined. Do we have such hypnosis and blind faith in our doctors simply because of their white coats that we believe they are infallible? And, in turn, do they have such blind faith in the medical journals recommending a given new wonder medicine or vaccine that they rush to give the drugs or vaccines without considering these deeper issues?
I conclude this article, as I do many of my essays here at SBM, with a call to action. These problems can easily be fixed. They do not require new resources or even great sacrifices. They just require a change in standard practice. Universities, journal editors, regulators, and professional organizations can make it happen. In many cases they know exactly what to do — they just need to do it.

Authority versus science on integrative medicine
David Gorski on November 2, 2015
Link


David Katz doesn’t much like us here at Science-Based Medicine. In fairness, I can’t say that I much blame him. We have been very critical of his writings and talks over the years, dating back as far as Steve Novella’s deconstruction of one of Dr. Katz’s more infamous statements about using a “more fluid concept of evidence” to Kimball Atwood’s characterization of his tortured logic to my pointing out that his arguments frequently boil down to a false dichotomy of either abandoning science or abandoning patients.
Last week, Jann Bellamy did her usual great job discussing an unfortunate special supplement of the American Journal of Preventative Medicine (AJPM) entitled Integrative Medicine in Preventive Medicine Education. This supplement included articles summarizing the results of project called IMPriME (Integrative Medicine in Preventive Medicine Education), funded by the Health Resources and Services Administration (HRSA), to advance the inclusion of “integrative medicine” in Preventive Medicine residency programs. Not surprisingly, this project was led by Dr. Katz. Jann used this special issue as a jumping off point to show, quite correctly, how, when it comes to so-called “integrative medicine,” it is always about the “potential,” which has always been elusive and has never been realized. Unfortunately, the elusiveness of the amazing potential attributed to “integrative medicine” (formerly referred to as “complementary and alternative medicine” or “CAM”) has done almost nothing to dampen the ardor of its cheerleaders for “integrating” as much woo as they can into medicine, which is why a major journal would allow someone like David Katz to edit a special issue dedicated to articles discussing IMPriME’s findings.
Thanks to Jann’s post, it appears that Dr. Katz is most displeased with us again here at SBM. To express his displeasure, he has rattled off a little rant over at his usual non-academic hangout and quack-friendly Internet outlet, The Huffington Post. There, he castigates us with a post entitled Science and Medicine, Fools and Fanatics: The ‘Fluidity’ of Woo. Yes, right off the bat, it’s the same old strategy, to paint advocates of “integrative medicine” as the “reasonable” ones while those of us who object to integrating prescientific quackery into medicine are clearly the “fanatics” (or, if you prefer, the fools). In it, as usual, Dr. Katz lays down some real howlers in defense of his integration of woo with medicine.

Respect my authoritah! I’m a real scientist, dammit!
Many species of animals (humans included) engage in various agonistic behaviors. One form of such behavior is known as threat display and consists of the animal doing something to show its strength or make it look larger and more dangerous in order to intimidate potential opponents and thereby win without having to fight. For example, when threatened the frilled-neck lizard engages in an impressive display in which it raises its body, gapes its mouth to expose a bright-colored yellow lining, and spreads out its frill showing bright orange and red scales. It’s a reaction used to assert territory, intimidate predators, and impress females. Male gorillas howl and beat their chests to accomplish the same goals. The way Dr. Katz starts his article reminds me very much of an academic equivalent to the display of the frilled-neck lizard in the way he puffs himself up and tries to assert his academic dominance over us by showing us his science bona fides. Substitute the size of one’s CV for the impressiveness of the neck frills, and you get the idea.
Boasting how he was a “logical choice” to lead IMPriME and clutching his pearls mightily at the very thought that mere peons like us would have the temerity to criticize such an initiative led by such a bold visionary as himself, much less l’il ol’ us, Dr. Katz sarcastically noted, “Predictably, no sooner had the electrons announcing this settled into the Medline index, than the clique calling itself ‘Science-based Medicine’ weighed in, with customary disparagements,” before puffing up the frill on his neck thusly:
That this suggestion [of a more “fluid concept of evidence”] was parlayed into allegations of quackery, or flirtations with it, has always amused me when it hasn’t annoyed me, and struck me as absurd in every way. For one thing, I make for quite a bad quack. I have run a federally-funded clinical research lab for nearly two decades. My colleagues and I have contributed nearly 200 peer-reviewed papers to the literature, some few of which demonstrate the utility of certain “alternative” medicine modalities, some few of which suggest the futility of others. Along the way, our efforts spawned a novel technique, called evidence mapping, later adopted by the World Health Organization for other applications, which showed the variability of evidence underlying the domain of so-called “complementary and alternative” medicine.
My passion for science is rather an open book, as are my musings on what “holistic” medicine is, and should be. Speaking of books, I have authored one textbook exclusively about evidence-based medicine and its relationships with both research methods, and clinical decisions; and multiple editions, in the company of colleagues, of an epidemiology text much devoted to that same domain. I will protest no further in that regard.
Lately, Dr. Katz seems to begin his rebuttals to any criticism of his advocacy for integrating quackery with medicine this way, with an appeal to authority—his authority. Yes, Dr. Katz. We get it. You’re freakin’ awesome! You’re so much more a real scientist than any of us because of your couple hundred publications, your federal grants, and your “passion” for science. (Or so you say.) In contrast, what have I to offer, a mere peon with only one federal grant at present for a preliminary study and a fraction of the number of publications? Heck, I’m not even a full professor yet (although, barring any unforeseen glitches in the process, that situation should be rectified by spring). All I can offer are science, critical thinking, and snark, in that order. Fortunately, for arguments as bad as David Katz’s, they are more than enough. After all, it is the quality of evidence, science, and argument that matters. That’s why I rarely dwell long (if I mention them much at all) on my own qualifications, even when I’m writing about topics that fall under the purview of my specialty that I might reasonable be considered to be an expert in, such as screening mammography. As I’ve said before, doing so is just…unseemly to me. In my experience, it’s almost always an indication that the arguments to come will be weak. Even if the experience that leads me to that rule of thumb is mostly confirmation bias, Dr. Katz’s article does nothing to counter that experience.
Besides, given the rise of quackademic medicine, defined as the infiltration of quackery into academic medical research and practice, coupled with the existence of the National Center for Complementary and Integrative Health (NCCIH), formerly known as the National Center for Complementary and Alternative Medicine (NCCAM, a much nicer abbreviation), having federal grants and publishing a couple of hundred papers are no guarantee of being quackery-free.
Basically, here Dr. Katz is telling us, like Cartman on South Park, to respect his authoritah. Don’t get me wrong. I do respect academic authority. I’m just not overawed by it. I also don’t think it’s wrong to appeal to the authority of someone like Paul Offit on vaccines as a shorthand for having to go through all the evidence. What I respect far more than authority, however, are science, critical thinking, and compelling arguments. It bothers me little or not at all that I’m frequently attacked for not having being a “real” expert in scientific disciplines I discuss, and it’s usually not even other physicians doing it but lay people with no training in those disciplines themselves. In other words, David Katz needs to spend more time burnishing his arguments and less whining about upstarts who dare criticize his genius.
Starting with a reasonable premise and then using it to justify pseudoscience
To be fair, Dr. Katz does make a couple of reasonable points. The problem is that, as he has done so often in the past, he takes those points and drives right off the cliff of woo with them. For instance, Katz’s infamous 2008 quote about resorting to a “more fluid concept of evidence” would have sounded a whole lot less quacky if he hadn’t yoked it to a patient anecdote in which he took the advice of naturopaths and referred the patient to them, where he was treated with homeopathy, The One Quackery To Rule Them All, using the anecdote to illustrate why he thinks physicians must be willing to use that more “fluid concept.” It would also be less objectionable if he didn’t also construct a false dichotomy wherein if a physician is not willing to consider pseudoscience like homeopathy when he reaches the end of what science-based medicine can do for a patient, then he is “abandoning” the patient.
Referencing one of the articles he wrote for the AJPM supplement he edited, Dr. Katz argues:
Moving on to the “fluidity” that has proven such a diverting target for the slings and arrows of my science-based colleagues, what did I mean? Well, that, too, is a matter of the published record — previously, and again in the new compilation in AJPM. Colleagues and I proposed, based on years of wrestling with complex patients, many of whom, urgent medical needs still insufficiently addressed, had tried and exhausted all of the well-supported, conventional treatments, that evidence traversed 5 key considerations. Those include: what is known about a treatment’s safety; what is known about a treatment’s efficacy; how well those first two are known (i.e., the clarity of evidence); the patient’s preferences; and, importantly, the availability of other, untried treatments for the condition in question.
It’s hard to argue with too much of this; that is, up to a point. This is the sort of thing that science-based medicine does all the time, particularly with rare conditions for which gathering “gold standard” randomized clinical trial evidence for treatments is difficult or even nearly impossible. It’s also the sort of thing science-based medicine does all the time with pretty much every patient for which science-based treatments might be lacking or far less than ideal. It is not Katz’s contention here about the “fluidity” of science that is the problem. Clinicians who wrestle with complex patients on a daily basis are more than familiar with such questions, to the point that Katz’s argument above is basically trivial. Besides, it’s not the point above that we object to; it’s how it’s used to justify “integrating” quackery like homeopathy into science-based medicine.
For instance, Katz then writes:
One need not be a clinician, wrestling with especially challenging patients or otherwise, to see the relevant spectrum of decisions. Whenever a treatment option is unsafe, ineffective, unclearly supported by evidence, unpreferred by the patient, and/or surrounded by other, ostensibly better treatments for the same condition — it is time to move on, be that treatment a product of tree leaf, or test tube.
Who could disagree with this? Certainly not me. Well, not quite. Here’s one problem. Katz clearly doesn’t move on when a treatment is clearly unsupported by the evidence; e.g., homeopathy. Here is a physician who, as I keep harping on, used the “fluidity of evidence” as the justification for sending a patient to naturopathic quacks and letting them treat him with The One Quackery To Rule Them All. Here is a man who is quite comfortable working with naturopaths, whose specialty is a cornucopia of mutually contradicting pseudosciences and quackery.
Also, notice the last two conditions, if the treatment is “unpreferred by the patient, and/or surrounded by other, ostensibly better treatments for the same condition,” then it’s time to move on. As I’ve said time and time again, a competent adult can choose or refuse any treatment he or she wants, but if a patient refuses a validated science-based treatment modality it is not a valid justification for a physician to use quackery. Similarly, the lack of “ostensibly better treatments” for a condition is also not an excuse to invoke quackery. Yet that is exactly what Katz does, although he denies it to high heaven. For instance, look at this table from the article he references (click to embiggen):
Again, this sounds superficially reasonable. The problem comes in how CAM/integrative medicine enthusiasts define the word “possible” under the Efficacy column and the word “unclear” under the Science column. Once again, the science for homeopathy is not “unclear,” nor is its efficacy “possible.” For instance, later in the paper, Katz argues:
Recent outcomes research on models (or “whole systems”) of integrative care32, 33, 34, 35, 116, 117, 118 demonstrate promise and innovation, as well as potential cost savings,8, 119 despite a lack of large-scale funding160, 161 and methodologic challenges.162, 163, 164
I note that most of the “whole systems” references cited by Katz refer to naturopathy and traditional Chinese medicine. As before, I point out that naturopathy is not only a cornucopia of quackery but that homeopathy is part of naturopathic training to the point of being included on the naturopathy licensing examination (NPLEX), and that what we now refer to as “traditional Chinese medicine” is the result of a rather cynical retconning of Chinese folk medicine by Chairman Mao Zedong in the 1950s. Indeed, 150 years ago, acupuncture was indistinguishable from bloodletting.
In which Dr. Katz lights a straw man about medical plausibility on fire
As you can see above, if your criteria for determining whether the science is “unclear” or the efficacy of a treatment is “uncertain” are lax enough, you can use Dr. Katz’s criteria to justify trying almost anything on a patient. Faith healing, anyone? Reiki? Well, it turns out that Dr. Katz does not accept that energy healing modalities are incredibly implausible. In what has to be the most embarrassing passage in the entire article (at least I’d be embarrassed if I had written it), Katz declares:
What is most fascinating and relevant about [E.O.] Wilson’s account is not the technology that is now revealing to us realities to which we were insensate before: colors we could not see; sounds we could not hear; energy fields we could not feel. What is most compelling is that all of these ARE perceivable to biology, just not OUR biology. There are animals that see colors we can’t; animals that hear sounds we can’t; and animals that feel one another’s electromagnetic fields. This, of course, is not woo; it is established biological fact.
The sounds, the colors, and the fields exist; and biologies other than our own can access them. Given, then, the range of human aptitude — from Helen Keller to the Rain Man; Michelangelo to Mozart; Edison to Einstein — are we truly prepared to say that no human being could possibly perceive the energy field produced by our bodies? Are we prepared to know in the absence of evidence that no such person endowed with a heightened sensitivity blunted in the rest of us, could interact with the energy field produced by a body in close proximity?
My radical version of alleged woo is captured in my answer to these questions: no. I simply don’t know what we don’t know. No one else does, either. No one else, that is, save fools and fanatics.
This is a straw man so massive that the heat derived from burning it could be used to generate electricity for the world for decades, much like Katz’s hot air boasting about his credentials. No one says that it’s impossible that a human being might be able to perceive energy fields. We say that it is incredibly improbable based on what we know about the basic sciences of biology, physics, and chemistry. It’s the same reason that I never quite say that it’s absolutely impossible that, for example, radio waves from cell phones might influence the development of cancer. I just know, based on cancer biology, chemistry, and physics, that it is so incredibly implausible that radio waves cause cancer, of the brain or breast or other organ, that it is reasonable, barring new evidence, to provisionally treat it as impossible. It is then up to believers like Katz—and, make no mistake, he is a believer in CAM, his protestations of being all about the science notwithstanding—to provide the evidence that a phenomenon like humans who can detect and manipulate other humans energy fields actually exists. It would also help if they could actually demonstrate that a human energy field, of the sort claimed to be manipulated by energy healers and TCM practitioners, actually exists, which they can’t. Given that, for example, for homeopathy to be true huge swaths of conventionally understood physics and chemistry would have to be radically rewritten, there would be many Nobel Prizes involved in such a discovery. The same is true of energy healing. Why hasn’t the evidence been found? Could it be because this phenomenon doesn’t occur?
At this point, it is obligatory, and rightfully so, to show that the hypothesis that a human being can detect another human’s “energy field” is rather trivial to falsify, so trivial that even a 12-year-old girl can do it.
Science doesn’t know everything, and skeptics criticizing woo are akin to the Spanish Inquisition
Circling back to Jann’s article that so riled Dr. Katz up in the first place, I note that her central idea was that, because so much of CAM and “integrative medicine” is based on pseudoscience and quackery, there is little or no evidence of its efficacy for the indications claimed, which leaves its apologists nothing but pointing to the “potential” of CAM. Flowing naturally from that central idea is the observation that that “potential” has never been realized—and almost certainly will never be realized—because there’s no “there” there. CAM is all either pseudoscience or the co-opting and rebranding of perfectly science-based disciplines like pharmacognosy or “nutrition” as being somehow “alternative,” so whatever “potential” CAM might have would come from such rebranded treatments as being CAM rather than medicine. Even then, the results have been disappointing, claims that the recent Nobel Prize in Medicine somehow “validates” traditional Chinese medicine notwithstanding.
So what someone who declares himself “passionate about science” to do when science doesn’t support his view on medicine? Easy! Pull the “science doesn’t know everything” gambit (a.k.a. an appeal to future knowledge or an appeal to future vindication) by comparing yourself to great scientists of the past whose discoveries overthrew old scientific paradigms:
That the proposition — the fluid nature of evidence, and more importantly science, should be respected — is contentious in the first place is, frankly, bizarre. I trust I am reliably conjoined to my science-based confrères (they are, in fact, male predominated so far as I know, and whatever that implies) in noting that the greatest of scientific minds — Einstein and Hawking and Newton; Darwin and Dawkins; Copernicus and Galileo and Herschel — were exceptionally devoted to the fluidity of science. They have, and in some cases still do, challenge the conventional understanding. They respect what we know, and how we know it, but humbly concede the potential for it to change as new evidence accrues.
This is, of course, another massive straw man. Perhaps Dr. Katz could point me to a passage—any passage!—in the entire nearly eight year history of this blog that could be construed as saying that science isn’t fluid or that there are findings in science that are absolute and not subject to change. I can’t find it. However, there are some parts of science that are—shall we say?—more fluid than others. Out near the “cutting edge” of science, findings and understandings are very fluid; back towards the core, less so. In other words, it’s a lot less likely that central laws, such as the three laws of thermodynamics, will be overthrown than it is that more recent and tenuous findings will be. Science is not a monolith. Besides, major paradigm shifts build on the old; they seldom completely replace it. My favorite example of this is how the theory of relativity supplanted Newton’s laws. Of course, it didn’t really supplant Newton’s laws, but rather improved on them. Newton’s laws are, in fact, still used for most calculations of motion used in aeronautical engineering because relativistic corrections are not necessary until velocities reach a fraction of the speed of light sufficiently large to exceed the precision required for a task. Basically, at most speeds that humans can observe and use, the theory of relativity reduces to Newton’s laws. That’s how science works.
Unfortunately, for something like energy healing or homeopathy, both of which Katz has advocated, new science much more revolutionary than the theory of relativity replacing Newton’s laws would be required to make them plausible. Barring that, Katz’s likening CAM investigators to Einstein, Hawking, Copernicus, et al is risible at best. Apparently, Katz views himself and his fellow integrative medicine enthusiasts as scientific groundbreakers who are unappreciated. I’m reminded of what Carl Sagan once said: “The fact that some geniuses were laughed at does not imply that all who are laughed at are geniuses. They laughed at Columbus, they laughed at Fulton, they laughed at the Wright brothers. But they also laughed at Bozo the Clown.” I’m surprised at this point that Dr. Katz was able to restrain himself from using Arthur Schopenhauer’s dubious quote about how “all truth passes through three stages.”
Of course, it’s not enough to appeal to “science doesn’t know everything” and imply that someday Katz and his fellow CAM enthusiasts will be viewed as scientific pioneers (a.k.a. the fallacy of future vindication). They must also paint their critics as close-minded and dogmatic, and what better way to do that then to write:
No one ever expects the Spanish Inquisition. Monty Python said it; it must be true.
And perhaps there is real wisdom in it. Perhaps the Spanish Inquisition is intrinsically shielded from all anticipation, because it is so improbable, even paradoxical. In this lamentable, historical episode, people were tortured and martyred for the heresy of their absolute faith in unknowable things, by people with absolute faith in unknowable things.
That’s why nobody expects the Spanish Inquisition. The martyrs and their murderers were too much alike, subject to differing versions of comparable hooey.
We have the inherent fluidity of scientific evidence to thank for every Nobel Prize in science. We have fluid thinking to thank for the innovations of Copernicus and Galileo; Darwin and Newton; Einstein, Edison, and Jobs. We have ideological rigidity to thank for the Crusades, the Salem Witch Hunts, 9/11, and the Spanish Inquisition. This is not a difficult comparison.
Yep, to Dr. Katz, we here at SBM and our fellow medical skeptics elsewhere are the equivalent of the Crusades, Salem Witch Hunts, 9/11, and the Spanish Inquisition. I’m surprised he didn’t mention the Holocaust as well or maybe Josef Stalin’s or Pol Pot’s atrocities. (Come on, Dr. Katz, do it! Compare us to Hitler! You know you want to! You know you want to so badly!) In any case, it’s true that this is not a “difficult” comparison to make. It’s all too simple, in fact! Actually, it’s so simple that it’s simple-minded. The ham-fisted way Katz explains the comparison also inadvertently conflates science and religious belief. After all, Katz refers to people being “tortured and martyred for the heresy of their absolute faith in unknowable things, by people with absolute faith in unknowable things.” In other words, both the persecuted and the persecutors believed in things that had no evidence and couldn’t be proven. Is Katz implying that all those scientists he lists underwent adversity for their faith in something unknowable? Likely not, but it sure sounds that way from what he wrote.
It’s sloppy thinking and writing, followed by even more sloppy thinking when Katz compares a scientific finding using poliovirus to treat cancer to homeopathy because it’s “using a toxin to treat a toxin.” No, it’s nothing at all like homeopathy. Cancer is not a “toxin,” nor is poliovirus. The engineered poliovirus does, however, release a toxin that kills cancer cells when it is in them.
I really have a hard time believing an actual physician made such a ridiculous analogy. Who’s the fool and fanatic here?
If you believe Katz, it’s us:
We need no help from fanatics to know there is hucksterism in the world, or bathwater in the tub. A common dose of common sense will suffice. The same measure will serve to renounce the counsel of fools to overlook the baby in the tub, or the possibility of somewhat surprising new truths in the fullness of time. We are most likely to find the advantages of wisdom in the realm of nuance and doubt; of open-minded skepticism; of caution, and hopeful curiosity.
The guardians will be there all along, telling us what is possible, and what isn’t — until it turns out it is. At which point, they will revise their fluid definition of woo, and pretend it never happened.
Oops. It looks as though I spoke too soon about Katz not invoking Schopenhauer, because that sure does sound like a paraphrasing of the ridiculous quote attributed to him about truth being “violently opposed” and then becoming “self-evident.”
Whenever someone like Dr. Katz invokes the “science doesn’t know anything” gambit, I like to quote Dara O’Briain:
For instance, around 1:50: “Science knows it doesn’t know everything. Otherwise it’d stop.” and “Just because science doesn’t know everything doesn’t mean you can fill in the gaps with whatever fairy tale most appeals to you.”
This particular O’Briain quote reminds me of Jann’s post again. In it, she quotes Katz’s definition of “integrative medicine” from his AJPM article, which I’ll expand:
Integrative medicine, a concept developed over the past few decades, refers to the fusion—by various means, and to varying degrees—of conventional medical practice and some of the practices that fall under the complementary and alternative medicine (CAM) rubric.
Doesn’t this sound like he’s filling in the gaps in medicine with whatever fairy story appeals to him? It does to me.
Jann also notes that if it is true that integrative medicine = SBM + CAM, then what’s left of integrative medicine if CAM is left out. Medicine. Or, more specifically, conventional medicine plus whatever science-based medicine CAM has “rebranded” and co-opted as being somehow “alternative.” Either way, it’s not as though woo-prone doctors and scientists like Dr. Katz haven’t been searching for evidence validating CAM therapies for a long time now. By and large, they haven’t found it, which has reduced them to making the sorts of appeals to future vindication and future knowledge that David Katz has invoked in his rebuttal and comparing their critics to the architects of Crusades, the Salem Witch Hunts, 9/11, and the Spanish Inquisition. I’m reminded of a suitably themed Geico commercial. It’s what you do when you believe in something but don’t have science on your side to support that belief.
Concluding this discussion, as part of the Spanish Inquisition, I sentence Dr. Katz to the comfy chair for his crimes against science and logic:
At least it might make him a little less angry at us.

So-Called Alternative Medicine
Edzard Ernst calls it “So-Called Alternative Medicine”. This insider’s view of SCAM is a new book from an prolific researcher and author.
Scott Gavura on June 14, 2018
Link

I have been a fan of Dr. Edzard Ernst for over a decade. His book Trick of Treatment, co-written with Simon Singh, was one of the first books I read on alternative medicine that gave me a perspective of just what “alternative medicine” was (or purported to be). As a pharmacist working occasionally in a drug store that focused on supplements and products like homeopathy, I knew these products lacked evidence of effectiveness, but had never really considered Complementary and Alternative Medicine (CAM) as a cult-like industry, or the double-standard from regulators that enabled all of these sometimes-absurd products to be sold in pharmacies. Dr. Ernst has written several books on alternative medicine, and Harriet Hall has reviewed most of them. Now Dr. Ernst has written SCAM: So-Called Alternative Medicine, a collection of 50 essays on topics related to SCAM. Each chapter is an essay, and each essay examines some aspect of SCAM. This book is not for true believers. If you’re a regular reader of this blog, then this book is for you.
Why “so-called” alternative medicine?
So-called alternative medicine (SCAM) goes by many names such as integrative medicine (which is really more of a brand than a specialty), holistic medicine, unorthodox medicine, and functional medicine. Tim Minchin describes it best, in my opinion. Ernst says that whatever you want to call it, it is not an alternative:
If a therapy does not work, it cannot be an alternative to medicine
If a therapy does work, it does not belong to alternative medicine, but to medicine.
Ernst is a physician and an academic, and his impressive credentials deserve a mention. A German by birth, he earned a medical degree and practiced in a setting where SCAM was fully embedded into conventional medicine. He was a homeopath and even worked in a homeopathic hospital. After spending time in a research position in Austria, he took the (world’s first) professorship in Complementary Medicine at the University of Exeter in England. Ernst built a research team and a research program, determined to critically evaluate SCAM with good science. His team was highly productive, conducting dozens of clinical trials and systematic reviews and compiling, over time, over 1,000 publications. But it was clear relatively early on that:
    • few forms of SCAM seemed to work
    • many seemed not to do more good than harm
    • and most SCAMs were so under-researched that it was impossible to tell
Ernst’s 2001 book, The Desktop Guide to Complementary and Alternative Medicine examined 700 treatment/condition pairs and concluded that only 7.4% were supported by good evidence. Realizing the vast gap between popular perceptions about the efficacy of SCAM and the reality, Ernst broadened his publishing targets from academic publications for health professional to eventually, consumer books like Trick or Treatment. Ernst prefers science over wishful thinking, and was surprised to find that SCAM enthusiasts felt otherwise. Ernst realized this early on in academic posting at Exeter:
On short notice, I had accepted an invitation to address this meeting packed with about 100 proponents of alternative medicine. I felt that their enthusiasm and passion were charming but, no matter whom I talked to, there seemed to be little or no understanding of the role of science in all this. A strange naïvety pervaded this audience: alternative practitioners and their supporters seemed a bit like children playing “doctor and patient”. The language, the rituals and the façade were all more or less in place, but somehow they seemed strangely detached from reality. It felt a bit as though I had landed on a different planet. The delegates passionately wanted to promote alternative medicine, while I, with equal passion and conviction, wanted to conduct good science. The two aims were profoundly different. Nevertheless, I managed to convince myself that they were not irreconcilable, and that we would manage to combine our passions and create something worthwhile, perhaps even groundbreaking.
During my short speech, I summarized my own history as a physician and a scientist and outlined what I intended to do in my new post—nothing concrete yet, merely the general gist. I stressed that my plan was to apply science to this field in order to find out what works and what doesn’t; what is safe and what isn’t. Science, I pointed out, generates progress through asking critical questions and through testing hypotheses. Alternative medicine would either be shown by good science to be of value, or it would turn out to be little more than a passing fad. The endowment of the Laing chair represented an important mile-stone on the way towards the impartial evaluation of alternative medicine, and surely this would be in the best interest of all parties concerned.
My audience, however, was not impressed. When I had finished, there was a stunned, embarrassed silence. Finally someone shouted angrily from the back row: “How did they dare to appoint a doctor to this chair?” I was startled by this question and did not quite understand. What had prompted this reaction? What did this audience expect? Did they think my qualifications were not good enough? Why were they upset by the appointment of a doctor? Who else, in their view, might be better equipped to conduct medical research?
It wasn’t until weeks later that it dawned on me: they had been waiting for someone with a strong commitment to the promotion of alternative medicine. Such a commitment could only come from an alternative practitioner. A doctor personified the establishment, and “alternative” foremost symbolized “anti-establishment”. My little speech had upset them because it confirmed their worst fears of being annexed by “the establishment”. These enthusiasts had hoped for a believer from their own ranks and certainly not for a doctor-scientist to be appointed to the world’s first chair of complementary medicine. They had expected that Exeter University would lend its support to their commercial and ideological interests; they had little understanding of the concept that universities should not be in the business of promoting anything other than high standards.
The content
Ernst divides his essays into several categories:
    • The Basics: What is SCAM? What is EBM? How much of EBM is actually evidence-based?
    • Common Problems in SCAM: The “root cause”, fear mongering, holism, conspiracy theories, and quackademia
    • Research and Researchers: Research in SCAM, promotion masquerading as research, NCCAIM, how to tell good from bad research
    • Healthcare Practitioners: Inside the brain of a SCAM practitioner, Doctors of Integrative Medicine, Veterinary Medicine, Nursing, Midwifery, Pharmacy
    • Patients and Consumers: Patient choice, when orthodox medicine has nothing more to offer, the wellness mania, alternative cancer cures
    • The Funny Side: Several humorous essays including How to become a Charlatan
Many SCAM advocates have accused Ernst of having a conflict of interest. He agrees he has a conflict, but not in the way his opponents claim. He notes:
    • evidence is a prerequisite for progress in healthcare,
    • evidence must be established by rigorous research,
    • we should not tolerate double standards in healthcare,
    • patients deserve to be treated with the best available treatments,
    • making therapeutic claims that are not supported by sound evidence is wrong.
Why SCAM research is rarely negative
Some SCAM journals and SCAM researchers never publish negative findings. This isn’t evidence of SCAM’s effect, but rather it’s the consequence of a trial design that cannot be negative. Ernst calls this the A vs. A+B design, and he describes it in a blog post studying a trial that purported to show benefit from acupuncture on cancer-related fatigue. Patients received “usual care” or “usual care” plus acupuncture. The acupuncture group did better. Ernst notes this design is designed to show an effect:
Imagine you have an amount of money A and your friend owns the same sum plus another amount B. Who has more money? Simple, it is, of course your friend: A+B will always be more than A [unless B is a negative amount]. For the same reason, such “pragmatic” trials will always generate positive results [unless the treatment in question does actual harm]. Treatment as usual plus acupuncture is more than treatment as usual, and the former is therefore more than likely to produce a better result. This will be true, even if acupuncture is no more than a placebo – after all, a placebo is more than nothing, and the placebo effect will impact on the outcome, particularly if we are dealing with a highly subjective symptom such as fatigue.
The Tricks of the SCAM Trade
One of my favourite essays describes some of the tactics used by SCAM advocates to promote their services.
Treat a non-existent condition: There’s nothing better than treating a condition the patient doesn’t even have. From the chiropractor’s subluxation to the acupuncturist’s chi blockage, these diagnoses are made up by the practitioner, and have no basis in reality.
Maintenance treatment: Persuade the patient who is healthy that they need “maintenance” treatments to stay healthy, such as IV vitamin infusions or the chiropractor recommending regular adjustments for spinal alignment. To the worried well, this may sound attractive, but it isn’t based on any objective benefit.
Things must get worse before they get better: Patients may feel nothing, or even worse, after a SCAM treatment. Practitioners may describe this as a “healing crisis”, an imaginary phenomena. In the case of conditions that are chronic or wax and wane, the practitioner can attribute worsening to the “healing crisis” and claim any improvement as evidence of effectiveness.
A cure takes a long time: To continue SCAM treatments despite a lack of any improvement is hard to justify on medical grounds, but practitioners make this claim routinely.
The problem is due to conventional medicine: With a nod to the conspiracy theories that are endemic to SCAM, problems are often blamed on conventional medicine. You see this routinely in harms or circumstances attributed to therapies like vaccines.
Detox: A marketing slogan and not an actual treatment, CAM proponents claim detox will rid the body of non-specified “toxins” that conventional medicine fails to acknowledge or treat.
The test of time: Some SCAMs have been around for hundreds of years (or more). SCAM advocates point to this long history of use as evidence of safety and effectiveness. Ernst points to bloodletting as a treatment with a long history of use that was appropriately discarded when the human body became better understood.
Energy: Energy is one of those terms that means whatever the SCAM practitioner wants it to mean, but it’s usually a proxy for “vital force“. Energy just sounds more scientific. While vitalism has been discarded by science-based medicine, it’s at the root of many SCAMs, like naturopathy.
Stimulating the immune system: Demonstrating rather startling ignorance of immunity, SCAM “stimulation” is a placebo. Moreover, it’s probably something we wouldn’t want to do, if we could.
A good book for the fledgling skeptic or SBM enthusiast
If you read SBM daily and are familiar with Dr. Ernst’s work or his blog, this book covers familiar topics. As a non-academic, I enjoyed this book because it made me consider aspects of SCAM from the perspective of a researcher. Despite Ernst’s deep knowledge of this topic, he writes in an easily approachable style – this is not dense medical writing. This book is not likely to change the minds of any true believers – but nor is that the audience. If you’re interested in the role of science in medicine, and on how to sort out good from bad when it comes to health and healthcare decision-making, Ernst’s book is highly recommended.

[Full disclosure: Dr. Ernst’s publisher provided me with a free copy of his book.]

Andrew Weil Flirts with Evidence Based Medicine
David Weinberg on December 7, 2012
Link
Andrew Weil, MD, pops up quite frequently on SBM, most recently in this entry by Harriet Hall, so I will not spend much space introducing him. An excellent biography and critique of Dr. Weil was written by Arnold Relman, former Editor of the New England Journal of Medicine. It is over a decade old, but contemporary to some of the events described in this post, and still quite relevant.
Suffice it to say that Dr. Weil is one of the most successful and well recognized popularizers of alternative medicine. He has authored or coauthored dozens of books. His website sells everything from baby pacifiers to vitamins to breakfast sausages, packaged bearing his name and/or visage. He is an altmed rockstar. He has been a key player in the branding of alternative medicine. In particular, been an advocate of “integration” of traditional and alternative medicine. He has created and exported residency training programs, and more recently proposed board certification in integrative medicine.
I recently read a book entitled On Being Certain: Believing you are right, even when you’re not, by Robert Burton, nicely-reviewed and recommended by Harriet Hall. In his book Dr. Burton excerpted an interview with Dr. Andrew Weil, pointing out Dr. Weil’s profound certainty about the effectiveness of a particular alternative treatment in spite of contradictory evidence. Dr. Hall also discussed this section of the book in her review. I found the excerpts fascinating and decided to delve more deeply into the interview. I also found another interview with Dr. Weil relevant to his ideas about evidence.
An interview with Dr. Weil (among others) was incorporated as supplemental material to a PBS Frontline episode called: The Alternative Fix. The program was pretty well done, and prominently featured the skeptical views of emeritus SBM editor Wallace Samson, MD. This was the interview quoted by Robert Burton.
Early in the interview, while explaining the appeal of alternative medicine, he brandishes a common mantra of of the “alternative” community:
I also think that a lot of people seek out alternative practitioners in frustration. If their first choice were available, it would be to go to a medically trained person, an MD who was open-minded and had knowledge of things beyond conventional medicine and could advise them about how to use them.”
The implication, of course, is that those who accept the claims of alternative medicine possess the virtue of open-mindedness. This is demonstrably false, and Dr. Weil is about to prove it. Credulousness is not synonymous with open-mindedness. Later in the interview he relates a clinical observation:
I wrote up my experience with an old osteopath in Tucson, who was a master of method called cranial therapy. He would take a kid, one treatment of this very noninvasive, inexpensive method and they would never get another ear infection. I saw this again and again.
I have recommended in my writings on my website that kids with ear infections should go to osteopaths and get this method done.
Cranial manipulation fails by all science-based criteria. Think of it as chiropractic manipulation of the bones of the skull, only with less credibility. It is based on assumptions that have no basis in scientifically established anatomy or physiology. It also fails by evidence-based criteria. The British Columbia Office of Health Technology did a systematic review of craniosacral therapy, and found no evidence of efficacy.
Dr. Weil laments the fact that researchers and mainstream medicine ignored his observations:
After something like twenty years of trying to get the research community interested in this, we finally set up some tests of doing this with kids with recurrent ear infections.
Despite the lack of a plausibility and absence of supporting clinical data, the Office of Alternative Medicine, and it offspring, the National Center for Complementary and Alternative Medicine, came to his rescue and funded a randomized clinical trial of cranial manipulation (full text). The study was designed to investigate two therapies for the prevention of relapse in recurrent otitis media (middle ear infection) in children. In addition to cranial manipulation, they also studied the popular herbal supplement Echinacea pupurea. The methodology utilized a double randomized (factorial) study design. This is a legitimate study design that can answer 2 questions with one trial. 90 children with recurrent otitis media were enrolled. The study was blinded, to the extent that patients and their families were not informed of their randomized assignment. Half were randomized to cranial therapy. The control group received osteopathic examination without manipulation. The same 90 children were independently randomized to receive Echinacea or placebo.Before getting to the results of the study, I will provide some excerpts from an article about Dr. Weil written at the time the grant was awarded.
Weil would not specify a success rate that he considers optimal for the study, but said that if the results were not positive, he “would be reluctant'” to continue using these methods in his clinic….
If he does not get the positive results he anticipates, however, Weil said the structure of the study would probably be more to blame than the treatments themselves.
“Either the testing was wrong or we had the wrong practitioners,” Weil said. “The first question would be whether we designed the experiment the right way.”
He is so certain in the effectiveness of cranial therapy that he is preemptively impugning the quality of a study that would dare fail to confirm his faith in cranial therapy. This theme will be replayed, shortly.
The results of the study:
Interestingly, patients who received Echinacea actually did worse than patients randomized to placebo. Unsurprisingly, cranial therapy did not improve the outcomes of children with recurrent otitis media.
There is a very curious aspect to this study. In the paper Dr. Weil is acknowledged as one of 2 principle investigators (PI) on the study, but neither he nor his co-PI is listed among the authors. In general, the principle investigator(s) is the driving force and team leader for a research program. He writes (or at least supervises the writing of) the proposal for the research grant. If the grant is found deserving, the funding is in awarded in the name of the principle investigator(s). The PI should be the prime mover in the design and execution of the study, as well as the analysis and presentation of the data. It is quite irregular for a PI not to be an author on a publication of the primary results of a research study. According to Dr Weil, he worked for 20 years to accomplish this study. We can only speculate why he chose not to appear as an author on a paper.
One has to give a great deal of credit to Dr. Weil for his successful navigation of the scientific method. He made an observation: Children who receive cranial manipulation seem to have fewer episodes of otitis media.
He formulated a hypothesis: Cranial manipulation may be effective in reducing recurrences in children with otitis media.
He designed a study to test his hypothesis. The study was funded and performed. The results were published. All kidding aside, this was not a trivial accomplishment. Dr. Weil comments on the study in the Frontline intervwiew.
He concludes:
We were unable in those tests to prove that this [cranial manipulation] had an effect.
Gathering evidence and reporting data are critical activities, but the real reward of research is seeing new results become integrated into patient care. In the light of the results of a randomized clinical trial, how have Dr. Weil’s opinions about the benefits of cranial therapy for prevention of recurrent otitis media changed?
The problem is, I’m sure there’s an effect there. We couldn’t capture it in the way we set up the experiment.
Dr. Weil may have some legitimate grounds to be critical of the study: Was it adequately powered? Were study treatments done properly? Were treatment group assignments maintained? Were endpoints captured properly? It is the responsibility of an investigator to consider and discuss all of these things in critiquing his own study.
However…
Considering that the only evidence for the benefit of cranial therapy is an unpublished, self-selected, anecdotal series; when faced with the results of a negative randomized clinical trial, an objective, open minded person must at least concede the possibility that the study failed to document an effect because… it doesn’t work! Dr. Weil, however, is undaunted: “I’m sure there is an effect there.” He leaves no room for any other conclusion. He anecdotal experience outweighs the data from a randomized trial. The science must be wrong.
Now, armed with the data he awaited for 20 years, he must have been relieved to have some evidence to inform his advice regarding treatment of otitis media in children. In response to his own study, he has concluded:
…I can see no harm in recommending to people that they try cranial therapy from a qualified osteopathic physician, even though we still have not yet been able to verify this in a randomized controlled trial.…
I checked his website, and indeed, found this entry (accessed 8/28/12).
I have described the use of an osteopathic technique called craniosacral manipulation to end recurrent ear infections in children. I recommend this treatment highly.
He also advises putting warm oils into the ear canal, and opines:
Sometimes this treatment, combined with oral doses of echinacea, will also end a middle-ear infection, if you get on it at the very start.
Yes, despite his own study suggesting that echinacea was harmful in otitis media, he is still promoting its efficacy on his website.
Proponents of science-based medicine are often accused of being closed minded for eschewing implausible and unsubstantiated treatments. It is implicitly or explicitly stated that altmed practioners and promoters are open minded. Neither statement is necessarily true, or false. The fact is, one cannot define whether an individual is open minded or close minded based on a snapshot of an opinion on any given subject at any particular time. One must analyze how they came to reach their opinion and how they adjust their opinions base on new information. Does their opinion change when presented with credible, contradictory evidence?
In this instance, Dr. Weil has demonstrated unwillingness to change his opinion or his behavior based on new information. Even when the evidence, his own evidence, betrayed his hypothesis, he was unwilling to acknowledge even the possibility that he might be wrong. He has continued to promote an implausible treatment which failed to demonstrate benefit in a randomized trial, and even continues to recommend a treatment that was associated with outcomes worse than placebo.

Taking On the Wellness Industry
The wellness industry is just one more manifestation of quackery and pseudoscience in health.
Steven Novella on June 12, 2019
Link

The “Wellness” industry has an Orwellian name – the name itself is propaganda and flies in the face of reality. In fact, there is no reason to believe that the wellness industry improves anyone’s health. It is really a snake-oil industry selling lies and misinformation, and largely targeting a vulnerable population – women. Women are vulnerable in this context only because they have been the target of decades of cultural conditioning.
A recent commentary by Jessica Knoll in The New York Times makes a compelling case for this conditioning. She makes the argument, largely from personal experience, that “wellness” is mostly a euphemism for being thin. Women are disproportionately shamed for their bodies, and made to obsess over their weight and eating habits. For sure this is culture-wide and extends to men also, with specific vulnerable subpopulations including athletes, performers, and celebrities. But women are disproportionately targeted by our obsession with thinness. This may explain why they are disproportionately targeted by the wellness industry.
Knoll argues that eating “clean”, detoxing, avoid “bad” foods, and all the magic snake oil is really about losing weight. Of course, much of the promotion of snake oil is explicitly about weight control, but she argues much of the rest is implicitly about being thin as a marker of being healthy. Goop is now the ultimate expression of exploiting women by selling them all kinds of pseudoscientific nonsense claiming to promote “wellness”. But this phenomenon started long before Goop and extends well beyond it.
It’s interesting to consider the cultural threads that lead to and connect to the current wellness industry. I think the base of this pyramid, the underlying phenomenon that ties everything together, is simply quackery – marketing health claims, treatments, and products based on magic and pseudoscience. This is as old as history, and of course all medicine was essentially quackery until a few hundred years ago, and Western medicine was not fundamentally science-based until about a century ago. For most of human history quackery thrived for a few core reasons. One is that it is easy to sell people what they want to hear. Second, placebo effects can easily fool most people into believing that almost any treatment works. This generates anecdotes which can be marshaled to promote anything.
Over the centuries quacks have fine-tuned their craft. They know who to target and how. They weave sciencey-sounding statements into their claims, exploiting the public’s fascination with but mostly misunderstanding of cutting edge science. They exploit celebrity, pseudo-expertise, appeals to ancient wisdom, and health halos like being “natural”. Theirs is as much a craft of illusion as the stage magician.
It’s important to realize that the quacks were there first – before science, regulation, standards, ethics, and accountability infiltrated medicine. They were knocked back by the advent of modern medicine, arguably beginning in this country with the Flexner report of 1910, but they never went away. They simply adapted.
In addition there is also spa culture, the “wellness industry” of the 19th century. Spas go back to ancient time, originating as bath houses, but also hot springs and natural mineral lakes. These locations became popular for their alleged healing properties. Of course if you have people going to a location to get healed by special water, that is an opportunity for quackery waiting to happen. Formal spas evolved out of these natural spring locations, slowly adding other treatments to the hot baths. Carbonated drinks, in fact, were originally marketed for their healing properties and became part of spa culture, weaving in with herbalism. These cultural origins survive to this day, with products like root beer and ginger ale. These were tonics, only later repurposed as beverages.
This process continues to this day, with spas adding all sorts of pseudoscientific treatments, from reflexology to toxin cleanses. They are now inextricably interwoven with the wellness industry.
In modern times we now also have the alternative medicine industry. This is all just a rebranding of the quackery and spa culture of old. This was their response to science and regulation trying to take over medicine. They fought back by rebranding as “alternative” and trying to wrap themselves in the very science that was used to exclude them in the first place. This, unfortunately, worked spectacularly. But let’s make no mistake – alternative medicine is nothing but old-fashioned quackery dressed up in modern scientific trappings in order to game the system, to work their way into the modern system of regulation designed to protect patients from quackery.
Alternative medicine and “wellness” are a natural combination.
Another pillar is straight-up antiscience, like the anti-vaccine movement, anti-fluoridation, mercury fear mongering, Lyme quackery, and the like. These are all sub-movements within the greater alternative medicine movement. They all also share a feature in that they overlap considerably with conspiracy culture. There is also even overlap with movements not directly related to health, like environmentalism. The quacks have infiltrated this movement as well, promoting anti-science and conspiracy theories.
So now we have a massive cultural movement that includes spa-culture, wellness culture, the weight-loss industry (which is mostly redundant), conspiracy theorists, science deniers, snake-oil peddlers, and old-fashioned quacks – all working together under the giant umbrella of “alternative” medicine. Their collective goal is to chip away at the regulatory infrastructure designed to promote a standard of care and protect consumers. They are blurring the lines between science and pseudoscience, true and fake expertise, reality and conspiracy theories, academia and fake professionals.
Academics and scientists have generally failed to recognize this threat, and therefore have not adequately done their job of protecting the public. Politicians generally don’t have a clue. The public is left understandably confused. Social media was rocket fuel for this blurring of reality and promoting nonsense. And ironically, the evidence-based medicine movement gave inadvertent cover to pseudoscience by formally eliminating scientific plausibility from consideration when evaluating clinical claims.
These problems are all fixable, but I don’t think they will be anytime soon. And even if we did, quackery is not going away. It can only be minimized.
So yes, the “wellness” industry is a snake-oil industry, specifically targeting women and exploiting cultural pressure to be thin. But this is just the tip of a very large and very old iceberg of quackery.

   Misleading Language. the Common Currency of “CAM” Characterizations. Part I
Kimball Atwood
Link
The Best Policy
From time to time I have been reiterating that correct use of the language has much to do with logic; I should add that it entails also honesty. I use the word “honesty” in its broadest sense…
Concision is honesty, honesty concision—that’s one thing you need to know.
—John Simon. Paradigms Lost: Reflections on Literacy and its Decline. New York, NY: Clarkson N. Potter, Inc.;1980. pp. 48, 52
Prologue
In 1983, a naturopath in Alberta inserted balloons into the nostrils of a 20 month-old girl and inflated them. The child died of asphyxiation. Subsequently, a judge described the treatment—dubbed “bilateral nasal specific” by the chiropractor who had invented it—as “outright quackery.” [1] Fast-forward 15 years: a woman presented to the otolaryngology clinic at the University of Washington in Seattle “complaining of severe midface pain and epistaxis” (nosebleed). She had suffered nasal septal fractures caused by a similar treatment, by then renamed “NeuroCranial Restructuring” (NCR). In their case report, the surgeons who had treated the woman at U. Wash discussed the claims of NCR and explained that the relevant anatomy predicts that it is implausible and risky. They also reported that it is expensive: “$2000 to $4800 for a standard course (of 4 treatments).” They concluded:
This case report of a complication after a CAM procedure called NCR highlights the wide range of treatment options available to patients. It is important for otolaryngologists to be aware of the spectrum of CAM therapies that patients may pursue and be aware of potential complications from these procedures.
An accompanying editorial used similar language.
How is it that in 1983 a judge could offer a concise summary of the essence of such a method, whereas scarcely a generation later 5 highly-trained medical doctors, even after presenting the sordid facts, could only obscure it with bland euphemism?
Language Distortions: why “CAM” requires quotation marks
I recall having first seen the term “language distortion,” referring to promotions and apologies for implausible medical claims, in a 1998 article written by our fellow blogger Wally Sampson: “The Braid of the ‘Alternative Medicine’ Movement.” [2] In offering explanations for the popularity of “CAM,” Dr. Sampson described the various motivating factors as being intertwined, as are the strands in a braid. One of those factors was
Propaganda and Language Distortion
We now see a new use of an ancient tool used by experts at manipulation of the public mind. Even the words “holistic,” “alternative,” “complementary,” “unconventional,” and “unorthodox” are invented euphemisms intended to mislead. They are benign terms covering a vast array of practices—most of them unproved, dubious, disproved, absurd, and fraudulent. Any politician knows one must find an enemy, even a straw one, to win elections. The term “slash, burn, and poison” was invented by Laetrile advocates to demean ethical cancer medicine, and it worked and it stuck.
In a strange twist of the braid, constructivist sociologist-historians of medicine in an “alternative medicine” journal have already turned the tables on our analysis of language distortion and accused rationalist scientists’ use of realistic terms like quackery, misrepresentation, and fraud of being merely prejudicial and biased. They call for more neutral terms to describe absurd methods like homeopathy. Thus the strands of constructivism and propaganda complement each other in the braid.
In addition to the language distortions applied to the bizarre treatment described above, there are numerous other examples in the world of “CAM.” They follow a strikingly similar time course:
In 1989, “chelation therapy” for atherosclerosis made the FDA’s list of Top 10 Health Frauds. By 2001, the NIH was calling the treatment a form of “alternative medicine.”
In a series of editorials in the 1970s and ’80s, the editor of Archives of Internal Medicine and Chest called chelationists “pseudoscientific zealots” who were “abusing a precious freedom.” By 2001, an NIH protocol referred to those zealots as “prominent experts.”
In 1984, Representative Claude Pepper convened a Congressional Hearing entitled Quackery: a $10 Billion Scandal. [3] Nine years later, an article in the New England Journal of Medicine trumpeted a similar dollar figure as the annual out-of-pocket expenditure by Americans for “unconventional therapies”—a collection of practices similar to those that had been portrayed as “quackery,” and their collective monetary cost as a “scandal,” by the Pepper Report. The lead author of the NEJM report later called the same practices “varieties of healing” and expressed his approval of the monetary costs by parlaying his findings into promotions of the practices and their practitioners.
In 1995, the Federation of State Medical Boards (FSMB) established a “Special Committee on Health Care Fraud.” In 1999, the committee’s name was changed to the “Special Committee on Questionable and Deceptive Health Care Practices.” In 2000, the committee appears to have vanished—to be replaced by the “Special Committee for the Study of Unconventional Health Care Practices (Complementary and Alternative Medicine).” That committee, aided by 3 consultants from academic “CAM” programs (one of whom was the first author of the NEJM report cited above), issued a report in 2002 that effectively reversed previous attempts to discourage physicians from practicing quackery. It also violated the FSMB’s formal policy of promulgating the Code of Ethics of the American Medical Association (AMA) for the purpose of “improving the quality of care…and appreciation of medical ethics” (to be cited next). The 2002 FSMB “CAM” report has since been accepted by numerous state medical boards.
Since at least 1977, the AMA Code of Medical Ethics has acknowledged that science-based medicine is not only a matter of competent practice, but also of honesty (quoted in part; emphasis added):
E-3.01. Nonscientific Practitioners. It is unethical to engage in or to aid and abet in treatment which has no scientific basis and is dangerous, is calculated to deceive the patient by giving false hope, or which may cause the patient to delay in seeking proper care.
E-3.04. Referral of patients. A physician should not so refer the patient unless the physician is confident that the services provided on referral will be performed competently and in accordance with accepted scientific standards and legal requirements.
The AMA has not re-worded those tenets. As we’ve seen, however, the FSMB and several state medical boards have. Here is how the FSMB “CAM” report—citing the National Center for Complementary and Alternative Medicine (NCCAM)—defines treatments that have no scientific basis:
Complementary and Alternative Therapies in Medical Practices (CAM)
CAM refers to a broad range of healing philosophies (schools of thought), approaches and therapies that mainstream Western (conventional) medicine does not commonly use, accept, study, understand, or make available. A few of the many CAM practices include the use of acupuncture, herbs, homeopathy, therapeutic massage, and traditional Oriental medicine to promote well-being or treat health conditions. People use CAM treatments and therapies in a variety of ways. Therapies may be used alone, as an alternative to conventional therapies, or in addition to conventional, mainstream therapies, in what is referred to as a complementary or an integrative approach. Many CAM therapies are called holistic, which generally means they consider the whole person, including physical, mental, emotional and spiritual aspects.
At the risk of belaboring the obvious, I’ll “deconstruct” that statement. The dominant theme is question-begging. “Healing,” “therapies,” and “to promote well-being or treat health conditions” assume a characteristic that the authors of the statement couldn’t have known to be true, i.e., that the methods in question actually work. That conclusion is readily demonstrable, even to the naive, by the statement’s having emanated from the NCCAM—thus identifying the methods as among those that the NCCAM has deemed in need of clinical trials.
“Holistic” also begs a question, albeit one that may escape the comprehension of the naive: homeopathy can’t be holistic, because it eschews pathophysiology in favor of “symptoms,” and for many other reasons; Traditional Chinese Medicine can’t be holistic because it knows nothing of the nervous system, the endocrine system, cells, anything at the molecular level, and more; chiropractic can’t be holistic because it clings to its “subluxation” theory and its theory of “nerve compression” leading to all manner of systemic “dis-ease,” against all evidence; naturopathy can’t be holistic because it readily accepts all three of the aforementioned claims, along with almost any other implausible claim that comes along, and ignores overwhelming evidence to the contrary. And so forth, for every “CAM” claim there is. “Holistic,” like many other terms and phrases common to “CAM” promotions, is an advertising slogan.
Another language distortion in the FSMB statement is the standard mislabeling of modern medicine by two devices: misrepresentation and derogation. Thus “conventional” is mildly derogatory, implying that modern medicine is boring and uncreative. “Western” is similarly derogatory in this era of neo-romanticism. It is also misleading, because modern medicine is no more ”Western” than the physics of Einstein is “Jewish.” The FSMB statement spares us the misnomer “allopathic,” which is ubiquitous elsewhere and is both derogatory and erroneous.
“Western (conventional) medicine” apparently displays its calcified resistance to innovation by its refusal to “commonly use, accept, study, understand, or make available [CAM].” Question-begging there, too.
Please indulge me now in contrasting the definition of “CAM” quoted above with a concise definition:
A spectrum of implausible beliefs and claims about health and disease. These range from the untestable and absurd to the possible but not very intriguing. In all cases the enthusiasm of advocates vastly exceeds the scientific promise.
What about the term “Complementary and Alternative Medicine” itself? Like the Moral Majority, it is neither. Need it be said that “alternative medicine” implies an alternative form of medicine that works? Many of us in the “CAM”-skeptic crowd have made fun of such misnomers, so there are plenty of useful examples to choose from and it’s easy to think of more. I’ll quote a couple that I’ve found amusing. This is from Bob Baratz, the president of the National Council Against Health Fraud:
To get to the National Institutes of Health in Bethesda from DuPont Circle I can take a cab or the Metro, I can walk or even ride a bicycle. These are true “alternative” methods of travel. However, no matter how hard I wish, or whatever I may choose to believe, I can’t ride a magic carpet or self-levitate to Bethesda.
Physicist John Farley takes on “integrative,” but his points are equally valid for “complementary”:
Suppose that the “integrative” approach were to spread beyond medicine, and were to be more broadly adopted by other disciplines in the sciences. The biologists would “integrate” creationism with Darwinian evolution, while the chemists would integrate alchemy into modern scientific chemistry. The geologists would integrate the belief that the world is only 6000 years old (and flat) with modern dating of rocks. Physicists would integrate perpetual motion machines with the conservation of energy and the laws of thermodynamics. And the astronomers would integrate astrology and astronomy. Of course, this is ridiculous. It’s not a good idea to integrate nonsense with valid scientic knowledge.
Bob Baratz might be wrong, however, according to guest blogger Mark Crislip; and our fellow blogger Steve Novella has shown how “alternative” thinking might be “integrated” into engineering. Who says this crowd isn’t innovative?
Epilogue
It is impossible to mention events such as those described at the beginning of this essay without considering other aspects of the “CAM” debacle. In a legal twist almost as bizarre as the “medical” treatment itself, the judge in the Alberta case sentenced the naturopath to only a $1000 fine and 1 day in jail. [1] I don’t know enough about the case to explain this, but I suspect that it was due to the naturopath having been a “licensed health care professional.” Edzard Ernst’s words ring true—”the most meticulous regulation of nonsense must still result in nonsense”—and so do mine: “The endorsement of such practices by a state naturopathic licensing board…demonstrates that licensing NDs doesn’t protect the public. Rather, it shields quackery from the scrutiny of the law.”
Is it really possible that there is a licensed health care profession that approves of inflating balloons in the nasopharynx to effect a “controlled release of the connective tissue tension to unwind the body and return it toward to its original design?” Yes, there is. It is “naturopathic medicine,” licensed in Washington and about 15 other states (soon in yours, if you don’t watch out). The method that killed the toddler was taught at least as recently as 2001 near Seattle at Bastyr University, the flagship of naturopathic “medical schools,” and licensed “naturopathic physician” Dean Howell offers to teach “NCR” in 7 days, for a mere $10,000, to naturopaths, chiropractors, medical doctors, osteopaths, and dentists.
Perhaps the ENT surgeons who treated the woman at the University of Washington can be forgiven for their language misdemeanors, which were minor compared to the grand larceny of truth perpetrated by their colleagues in the Department of Family Medicine:
The philosophical approach of naturopathic medicine includes prevention of disease, encouragement of the body’s inherent healing abilities, natural treatment of the whole person, personal responsibility for one’s health, and education of patients in health-promoting lifestyles. Naturopathic practice blends centuries-old knowledge of natural, nontoxic therapies with current advances in the understanding of health and human systems. The scope of practice includes all aspects of family and primary care, from pediatrics to geriatrics, and all natural medicine modalities…
And so on.
References
[1] Butler K. A Consumer’s Guide to Alternative Medicine. Buffalo, NY: Prometheus Books; 1992:82.
[2] Sampson W. The Braid of the ‘Alternative Medicine’ Movement. Sci Rev Alt Med 2;2:4-11 (Fall/Winter 1998)
[3] Quackery: A $10 Billion Scandal, Hearing before the Subcommittee on Health and Long-Term Care of the Select Committee on Aging, House of Representatives (98th Cong., 2d sess., 1984). Comm. Pub. No. 98-463

   Misleading Language: the Common Currency of “CAM” Characterizations Part II
Kimball Atwood
Link
Background
I promised readers the “Advanced Course” for this week, which undoubtedly has you shaking in your boots. Fear not: you’ve already had a taste of advanced, subtle, misleading “CAM” language, and most of you probably “got” it. That was R. Barker Bausell’s analysis of how homeopathy is “hypothesized to work.” In the interest of civility, let me reiterate that I don’t think of Bausell as a horrible person or an ignorant boor for having written that statement. Rather, I think of him as having been so steeped in the de rigueur “CAM” language distortions of the 1990s that he is largely unaware of their insidious power. I suspect too that he, like most of us who grew up when schools no longer stressed the rigors of English composition, has an underdeveloped sense of the relation between the craft of writing and the integrity of its content. That doesn’t excuse him from writing honest prose, of course.
Last week’s post cited blatant language distortions of “CAM”—euphemisms, slogans, and outright falsehoods—and some that were more subtle: question-begging, misrepresentation, and derogation. It would require a semester’s worth of seminars to delve into the overlapping categories of misleading “CAM” language, but here we can consider a few. Then, perhaps, we’ll engage in an amusing diversion—more about that at the end of this post.
Introduction to the Advanced Course
For the most part, what interests us are not the assertions of “CAM” practitioners themselves, whose misrepresentations tend to be simplistic and obvious. We are more concerned with statements by influential “CAM” apologists in academic medicine and government, because it is they who have the greater capacity to deceive the public. This was illustrated last week by quotations from the Federation of State Medical Boards. Such apologists are frequently somewhat nuanced in their misrepresentations, both because they are more clever than the “CAM” practitioners that they patronize, and because they must address their assertions to a more clever audience—namely, physicians and biomedical scientists whose skepticism of “woo” makes them wary.
Sometimes, however, statements by “CAM” practitioners and those emanating from influential apologists are one and the same, suggesting that the latter are ditzier than their status implies, or perhaps just lazier: witness the statement about “naturopathic medicine,” quoted in last week’s post, from the website of the University of Washington Dept. of Family Medicine (which itself quotes similarly inane language from the U.S. Dept. of Labor). Other examples, from WebMD (“Better information. Better health.”) and InteliHealth (The Trusted Source®), can be found here. InteliHealth has improved slightly since 2002; WebMD has not.
Some “CAM” language distortions are an inevitable result of Evidence-Based Medicine’s (EBM) incomplete definition of “evidence.” We have seen examples written by the “CAM Field” of the Cochrane Collaboration (“The reliable source of evidence in health care”), whose EBM-style reviews are widely respected. More examples can be found on InteliHealth, which now buys much of its “CAM” information from Natural Standard (“The Authority on Integrative Medicine”). Whether or not the authors of such reviews are aware of established knowledge and other external evidence, the arcane rules of EBM (no longer surprising to SBM readers!) give them “plausible denial.”
Bait and Switch
For several years beginning in the late 1990s, there were clusters of “CAM” articles in a few influential medical journals. They are chock full of misleading language, but this discussion must be limited to a few examples. In the fall of 1999, the late virologist Stephen Straus became the first Director of the nascent National Center for Complementary and Alternative Medicine (NCCAM). Like many in the new academic world of “CAM,” Straus seemed innocent of “CAM.” Regarding the script of his new role, however, he was a quick study. In June of 2000 he wrote an article for Academic Medicine:
Both as an art and as a science, medicine is ever-evolving. Before the emergence of the empirical and experimental sciences, physicians relied heavily on the art of medicine. But in the past century, progressively detailed investigations of individual organs, cells, and molecules led to the emergence of biomedicine, with its heavy reliance on science, as the now-dominant and conventional basis for medicine.
A superficially innocuous and uncontroversial, politely deferential (for the benefit of snickering ex-colleagues at the NIH, I’d guess) introduction. But how innocuous is it? In that short passage are the seeds of garden variety “CAM” language distortions: by sleight of pen, the “art of medicine” will quickly change to “CAM” itself and then to “healing arts,” after fleeting stops at “rich dialogue” and “the healing process,” only to return to “CAM.” ”Detailed investigations of individual organs, cells, and molecules” is a thinly-disguised reference to the objectionable “reductionist paradigm” of modern medicine, which by dint of its “heavy reliance on science” has overstayed its welcome.
The declaration that “biomedicine” is “the now-dominant and conventional basis for medicine” manages to “dis” modern medicine in ways that most physicians and scientists would hardly notice, but that the “CAM” crowd and its congressional patrons would hardly miss: “biomedicine” and “conventional” are mildly pejorative misnomers, as discussed last week; “dominant” is a potent adjective, intended to distract the reader from imagining that the emergence of modern medicine had something to do with the emerging understanding of nature over the past couple of centuries—and to replace it with the facile suggestion that medicine, like everything else, is a matter of power vs. oppression or privilege vs. exclusion. This kind of thinking had been fashionable in the academy for more than a generation, and was certain to win approving nods from such ex-hippie “CAM” luminaries as Andrew Weil, who authored an article for the same issue of Academic Medicine, and James Gordon, the Chairman of the White House Commission on Complementary and Alternative Medicine Policy. More from Dr. Straus’s piece:
Historical precedent predicts that a number of contemporary CAM therapies [will prove effective]…some of our most important pharmacologic agents…are derivatives of herbal products. Radiation therapy, once regarded as a radical approach to treating cancer, is now a standard of care, and acupuncture, considered arcane and primitive before Nixon went to China in 1971, is now routinely prescribed to manage pain.
First point: no, it doesn’t, and even if it did so what? Nothing in “CAM” consists of the rational screening, for useful medicinal properties, of substances found in nature (bait and switch). Second point (radiation therapy): What does that have to do with “CAM”? Second point (acupuncture): Routinely prescribed by whom? And even if it were so, it isn’t evidence. The counterpart to the “straw man” argument is the “false premise” argument in which the author asserts the existence of a premise that supposedly justifies the point he is about to make. Unlike the case of the “straw man,” the author approves of the premise; but like that of the “straw man,” the premise is either a figment of the author’s imagination or does not support his subsequent point.
Here is the obligatory, once-an-article skeptical posturing to reassure the reader that she is in the capable hands of the author and of the author’s like-minded colleagues:
Many people believe that if a product has been used for centuries, it must be effective, and if it is natural, it must be safe. To the contrary, there is mounting evidence that some CAM therapies expose patients to potentially toxic components or displace effective conventional treatments…St. John’s wort [and protease inhibitors, for example]…
That passage is particularly rankling for its hypocrisy. The evidence for the dangers of mixing herbs and HAART agents came from legitimate Institutes of the NIH, but not from the NCCAM. The NCCAM, on the other hand, through its funding of the Bastyr (Naturopathic) University AIDS Research Center, may have contributed substantially to the dangers of such herb-drug interactions, and certainly contributed to an ethically-dubious trial of “Garlic for hyperlipidemia caused by high [sic] active antiretroviral therapy (HAART).” Garlic is another substance that reduces blood levels of HAART agents. At about the same time that Dr. Straus must have been writing his article for Academic Medicine, personnel from Bastyr and the University of Washington were presenting an abstract in which they reported having temporarily stopped the garlic trial because of this new information, but “additional funding was later received from NCCAM for implementation of safety mechanisms to allow the trial to resume.” Those “safety mechanisms” did not include “therapeutic drug monitoring.”
More from the article:
Lacking such knowledge yet enamored of the appeal of CAM practices, the American people have demanded that these be studied.
The American people are neither “enamored of the appeal of CAM practices” nor have they “demanded that these be studied.”
The Wooification of Health Care
And now a cynical fest of question-begging, baiting and switching, and stated intentions that should make any reason-loving physician shudder:
…an especially vital role with respect to facilitating the integration of rigorously proven CAM approaches into the ongoing education of physicians and the daily practice of medicine…Dr. Andrew Weil…spoke of the need to train young physicians about CAM. Few would argue against the wisdom of this notion…Medical students should receive instruction about proven CAM methods…the curriculum should be enriched by exposures to the history of medicine, medical ethics, and medical economics…We also wish to work with the AAMC to overcome the reluctance of conventional physicians to consider CAM therapies that are proven safe and effective for their patients…in this way may we succeed in expanding the repertoire of ways to achieve and maintain health and restore an appropriate balance of both the art and the science of medicine.
We might be comforted by observing that there are no “CAM therapies that are proven safe and effective,” nor are there likely to be any; except that we already know that “CAM” researchers, encouraged by misguided shills for EBM, don’t see it that way—and neither, if we don’t speak out, may medical schools, patients, insurance companies, and governments. The passage is yet another language distortion, which means exactly what “CAM” advocates choose it to mean.
The point is made even more pointedly by at least two other articles in that issue of Academic Medicine. The first instance is no surprise, coming from Andrew Weil and his pals at the University of Arizona:
The addition of proven CAM to conventional medicine would be an improvement, yet this combination alone is not what could be defined as integrative medicine….Integrative medicine is defined as healing-oriented medicine that reemphasizes the relationship between patient and physician, and integrates the best of complementary and alternative medicine with the best of conventional medicine. We believe that this synthesis of humanistic medicine, patient- and relationship-centered care, preventive health, allopathy, and CAM is the model for creating an improved system of health care.
So: it isn’t even just something that the ditzy members of the class can elect; if Weil and his gang have their way, it will be mandatory for all—it became so at Georgetown a couple of years ago, as reported by Orac. Such training already exists for ”integrative medicine” fellows at U of Arizona and includes claims that don’t “fit with our Western paradigm,” scrapping RCTs in favor of “outcomes research,” asking “physicians to explore their beliefs about medicine and…the assumptions that underlie medical science,” preparing “physicians for new roles as partner, coach, teacher, motivator, and healer,” heightening “awareness of the different cultural healing systems,” “Chinese medicine, energy medicine, homeopathy, allopathic medicine, manual medicine, and Western herbalism,” and “physician self-care” that includes “therapeutic movement,”
facilitated sessions [for] examining [one's] humanity, including the identities they have left behind and the physician-healers they are seeking to become.
These meetings occur regularly, often offsite, and in addition to traditional group work, include experiences ranging from listening to music, hiking, and reading plays and poetry to participating in Sufi dancing and sweat lodge ceremonies.
Please pass the homeopathic Ipecac.
The final few quotations are from another article in that issue of Academic Medicine, co-authored by Ralph Snyderman, who at the time was the Chancellor for Health Affairs at Duke University:
Integrative medicine is an approach to the practice of health care based on a sound scientific approach with an emphasis on the responsibility of the physician to engage the patient in his or her own unique plan for health. Essential aspects include the recognition of the importance of the mind-body relationship and a willingness to consider unconventional modalities with informed skepticism and scientific evaluation. Above all, integrative medicine encompasses the caring bond between the patient and the caregiver, and the responsibility of the latter to enable the patient to benefit from a full array of modalities that can be shown to benefit health.
What’s so remarkable about that statement is just how unremarkable it is. With the possible exception of “the mind-body relationship” and “unconventional modalities” (which could easily be changed to “psycho-social considerations” and “innovative treatments” without a literal change in meaning), if “integrative” were changed to “modern” it would hardly stir the dozing medical students in the back row. That, of course, is exactly why its misleading language is so pernicious: it can’t really mean “based on a sound scientific approach” or “with informed skepticism and scientific evaluation,” because those would preclude what’s really meant by “integrative.” Sure enough, the rest of the article is a paean to Weil’s “Arizona Program,” which the authors intend to emulate at Duke, and to the Consortium of Academic Health Centers for Integrative Medicine, which is also committed to the wooification of modern medicine behind the veil of “scientific studies” (quoted in Snyderman):
Our mission is to help facilitate the transformation of health care through rigorous scientific studies, new models of clinical care, and innovative educational programs that integrate biomedicine, the complexity of human feelings, the intrinsic nature of healing and the rich diversity of therapeutic systems.
There are now 39 medical schools among the members of the consortium, including, in all likelihood, yours. As my colleague Dave Gorski has discussed, there are others that also push woo. Be afraid.
And now for the amusing diversion…
That was depressing, to say the least. As an antidote, I’m proposing a weekly “language distortion” game: the Weekly Waluation of the Weasel Words of Woo (W^5 for short). It’s simple: each week on Friday, I’ll post a passage at the end of my blog. Feel free to comment on its misleading language over the weekend (please stop before Monday and limit yourselves to one submission each, or I’ll get overwhelmed and confused), and I’ll select the really good ones and heap praise upon you the following Friday. Feel free to draw upon “external evidence.” Here’s the first entry, which calls to mind at least two recent SBM posts, here and here (hint, hint):
For people with chronic pain or with other refractory conditions, the chiropractic visit itself can be a source of comfort even without the addition of a demonstrable scientific component. Treatment by a chiropractor can generate a sense of understanding and meaning, an experience of comfort, an expectation of change, and a feeling of empowerment. Chiropractic’s combination of vitalist “innate intelligence” and simple mechanical explanation can give rich vocabulary for just those illnesses conventional medicine remains poorly equipped to address. Research indicates that for many of the illnesses chiropractic treats, precise diagnosis, assurance of recovery, and physician-patient agreement about the nature of a problem hasten recovery.
Chiropractic finds its voice exactly where biomedicine becomes inarticulate. Too often, biomedicine fails to affirm a patient’s chronic pain. Patients think their experience is brushed aside by a physician who treats it as unjustified, unfounded, or annoying, attitudes that heighten a patient’s anguish and intensify suffering. Chiropractors never have to put a patient’s pain in the category of the “mind.” They never fail to find a problem. By rooting pain in a clear physical cause, chiropractic validates the patient’s experience.
Happy Waluating!
Post-post addendum
At least two studies in recent years, both published in Academic Medicine, have shown that “CAM” courses in medical schools are overwhelmingly uncritical and promotional. One of these was done by our colleague Wally Sampson; the abstract is here. The abstract of the other is here.
A recent issue of Academic Medicine was once again devoted to the teaching of “CAM” in medical schools. A perusal of the table of contents and the available abstracts confirms that “CAM” advocacy is, more than ever, the norm for such ventures.

Inoculating – Against Misinformation
A new study confirms what we suspected – you can’t just correct misinformation with information, you have to expose the tactics of deception so people can recognize them for themselves.
Steven Novella on May 10, 2017
Link

This is perhaps the most “meta” post I have written for SBM. One core approach of scientific skepticism, of which SBM is part, is to use metacognition – thinking about thinking – to explore information and our understanding of that information. Sometimes, however, we need to think about how we think about thinking.
Specifically, one of our goals is to directly counter misinformation about science and medicine, misinformation that is often the product of marketing campaigns funded by multi-billion dollar industries. We definitely perceive this as a David and Goliath situation, and probably spend too much time debating about whether or not we are having a significant effect. Regardless, we are concerned with being as effective and efficient in our countering of misinformation as possible.
Fortunately, there is science to guide our promotion of science. How does misinformation change attitudes and how well do various strategies to counter misinformation work? A new study published in PLOS One (“Neutralizing misinformation through inoculation: Exposing misleading argumentation techniques reduces their influence“) explores this issue further and supports the basic strategy that we generally take at SBM. 
Neutralizing misinformation
There are two experiments in this study by Cook, Lewandowsky, and Ecker. The first is actually quite negative. They used as their subject matter the acceptance of anthropogenic global warming (AGW). The exact topic probably doesn’t matter and the principles should apply to medical topics as well. They exposed subjects to false balance about AGW, implying that there is controversy and no real consensus on whether or not AGW is real. In one group they just presented the misinformation and false balance. In a second group they first exposed them to a statement about the fact that there is a strong consensus for AGW. In a third group they first exposed subjects to information about the misleading effects of false balance, and in a final group they exposed subjects to both the consensus information and the false balance “inoculation” information. A control group was presented with no misinformation.
They also surveyed the subjects regarding their acceptance of free market values and various attitudes regarding AGW. After exposure to all the information in the various groups they surveyed them again regarding AGW. Overall the results were negative. There was no statistically significant effect on acceptance of AGW, attributing warming to human activity, for policies to address AGW, on trust in climate scientists, or trust in contrarian scientists.
There was a small but statistical effect on the perceive consensus. This is a weak effect – essentially telling people there is a strong consensus slightly countered later implying that there isn’t a strong consensus. Prior studies in which subjects were exposed to direct information showing a lack of consensus did not show this effect.
Attitudes predictably corresponded to free market values, again showing that ideology is the main determining factor with respect to accepting the science of AGW. Further, information about AGW had a polarizing effect, with strong free market supporters showing a “backfire” effect and rejecting AGW even more.
All of this is pretty disheartening. Misinformation + correcting information mainly had a polarizing effect on the attitudes of subjects, predicted by their free market ideology. I was not impressed by the small inoculating effect that was seen regarding the consensus specifically because the misinformation was only implied and not direct.
Fortunately there was a second experiment. This had a similar design, except that there were four groups, a control group with no intervention, a misinformation only group, an inoculation group, and an inoculation + misinformation. More importantly, the inoculation took a different form. Rather than just telling people correct information, or what false balance is, they explained to them how fake experts can be used to deliberately create the impression of false balance. In other words, they exposed the deceptive tactics used to sow distrust in the scientific consensus.
In this experiment also misinformation had a negative and polarizing effect on acceptance of AGW. However, in the group first inoculated with information about this deceptive tactic, the polarizing effects of misinformation were completely neutralized.
What does this mean?
I freely admit my bias here, because this study seems to support what I have been saying for more than two decades, and essentially my entire approach to science communication. At least when it comes to controversial topics, you can’t just give people information. You have to teach them metacognition – how to think about how they think about things. You need to teach them about cognitive biases, mechanisms of deceptive, logical fallacies, and the deceptive tactics of pseudoscience.
That is the niche we fill. There are plenty of outlets out there providing medical information. Mainstream medical academic groups do a fine job of analyzing, synthesizing, and communicating medical science about non-controversial topics. They do a generally horrible job, however, when it comes to pseudoscience, fraud, and misinformation.
This is because pseudoscience is a specialty unto itself – it is an expertise and knowledge base we possess that mainstream outlets lack. We specifically analyze the logic and tactics of those promoting pseudoscience in medicine and try to “inoculate” our readers by explaining how the tactics work and why their logic is flawed.
For example, it is not enough to hear the claim that the evidence shows acupuncture and homeopathy don’t work. It is more important to understand that proponents of acupuncture fudge their definition to make negative studies seem positive, or muddy the waters with mixed variables, such as “electroacupuncture.” It further helps to understand how proponents of both try to use pragmatic studies (not blinded and placebo controlled) to make efficacy claims, and why this is not valid (all of which were discussed by Dr. Gorski on Monday). We need to expose the strategy of presenting placebo effects as if they are real effects that support a specific intervention (they don’t).
We have done our job if our readers learn how to see a study and figure out for themselves why it doesn’t support the claims the authors make for it.
It’s nice to have some objective evidence that this strategy can actually work.
.

Infiltrative Pseudoscience
If you don’t think that CAM is the enemy of science in medicine, then you don’t understand CAM and its proponents. Don’t be fooled by their marketing. They want a return to the pre-scientific days when health gurus could sell any snake oil they want at exorbitant prices, with any hyped claims that they want, without going through all that tedious science. 
Steven Novella on January 17, 2018
Link

It’s always nice to see anti-quackery efforts in mainstream medical and science journals. We need much more of it. At times it can seem as if SBM advocates are a lone voice and the bulk of our profession are unaware shruggies about the threat of pseudoscience infiltrating our profession.
A recent article in The Surgeon by Ben Li, Thomas L. Forbes, and John Byrne was a rare and welcome sanity check pushing back against the recent fad of nonsense in health care. Their title, “Integrative medicine or infiltrative pseudoscience?” coins a nice term I will probably start using. That is exactly what so-called alternative medicine is – an infiltration of pseudoscience into the medical profession and health care industries.
The article will appear pretty basic for regular SBM readers, but that is OK. It is a good introduction to the problem, and anything that stokes a broader conversation is a good thing. They outline the risks of infiltrative pseudoscience:
The use of CAM may cause harm to patients through interactions with evidence-based medications or if patients choose to forego evidence-based care. CAM may also put financial strain on patients as most CAM expenditures are paid out-of-pocket.
Some CAM treatments are not inert. They can therefore cause direct harm, or may have interactions with evidence-based treatments that cause direct harm. Usually, for those who have not thought deeply about this topic, potential direct harm is the only concern that comes to mind when considering the potential downside of using pseudoscientific treatments. This is only the tip of the iceberg, however.
With a little thought people will also usually realize that even harmless treatments may delay or replace effective medicine, and this too is a source of harm. While this may seem obvious, there is also some objective data to support this conclusion. As David Gorski recently noted, a 2017 study of cancer patients found that using CAM cancer treatments was associated with a higher death rate. This likely was mostly due to delayed or refused standard treatment.
The authors also note the potential financial harm, which I think is more extensive than even we realize. Once a desperate patient is in the clutches of an alternative practitioner, promising side-effect free “natural” cures for serious ailments, it is not unusual for the mark to pay tens or even hundreds of thousands of dollars. Sometimes the patient or their family has to raise funds from friends, family, coworkers, and others to pay these high costs, spreading the harm around.
Since insurance companies are not involved, it may be difficult to carefully track these payments, which is partly why they are likely underestimated.
The authors also hit upon another harm, often missed because it is downstream and indirect:
Recently, there has been controversy surrounding the leaders of several CAM centres based at a highly respected academic medical institution, as they publicly expressed anti-vaccination views. These controversies demonstrate the non-evidence-based philosophies that run deep within CAM that are contrary to the evidence-based care that academic medical institutions should provide.
This, to me, is the real existential threat of CAM. Perhaps recent political events have made us generally more aware of how fragile our institutions can be. They really do depend on a culture of dedication to the underlying principles. If you erode that culture, no institution will save us from ourselves.
Our medical profession is premised on a dedication to science and evidence, arguably dating back formally to the Flexner report of 1910. But this dedication to science in medicine only exists so much as the individual people in the profession understand and are dedicated to a reasonable scientific basis for what we do. There needs to be a culture of science and standards.
The worst thing about CAM is that it erodes that culture. It is, in fact, a direct attack on that culture. CAM proponents seek to undermine the standard of care, to confuse how science works and the relationship between evidence and practice. They want to change the rules of evidence, to water down regulations or carve out exceptions, and to promote false ideas. For example, they have managed to convince much of the public that placebos are effective medicine, despite the fact that their very definition includes that they are not effective.
Expressing anti-vaccine views is just one tiny symptom of this deeper problem. It was good for drawing attention to the problem, but we should not think anti-vaccine views are the extent of the problem.
CAM also erodes the public understanding of science in medicine and public trust in medical science. This is arguably part of a broader cultural phenomenon of lack of trust in experts in general. I do wonder how much of the “death of expertise” was driven by the alternative medicine movement. It certainly contributed to this phenomenon.
I would also add to the list of potential harm of pseudoscience in medicine, redirecting limited resources away from more promising treatments and research. The very existence of the NCCIH (National Center for Complementary and Integrative Health) proves this point. The reason for this center is to provide funding for research into treatments that could not justify funding with proper science. As a result billions of dollars have been arguably wasted without resulting in a single new treatment, or diminishing the use of CAM by proving it doesn’t work. This is a small percentage of the overall NIH budget, but still that is billions of dollars of wasted medical research.
How many health care dollars have been wasted on demonstrably worthless treatments? What about paying for the harm of relying on those worthless treatments? How many patients have been diverted from actually promising clinical trials, and how many researchers have wasted their precious career time chasing fantasies?
I think it is clear, from the many articles we have published here on the topic, that if anything the medical profession needs to push for more science-based practices, higher standards in medical research and reporting, and better understanding of medical research. There is actually broad recognition that this is true. But simultaneously we are allowing a subculture of CAM that is pushing for the exact opposite, and too many of our colleagues don’t see the risk of this situation.
At the end of the day, our first professional duty is to our patients. The authors recognize this too:
Although there are financial incentives for institutions to provide CAM, it is important to recognize that this legitimizes CAM and may cause harm to patients. The poor regulation of CAM allows for the continued distribution of products and services that have not been rigorously tested for safety and efficacy.
That pretty much hits it on the head – don’t fall for short-term financial gain by selling snake oil. We have a duty to our patients to protect them from harm. This does partly stem from proper regulation, but that is not all. We need to affirm our profession’s dedication to a solid scientific basis for our practice. That means an abject rejection of everything that CAM stands for.
If you don’t think that CAM is the enemy of science in medicine, then you don’t understand CAM and its proponents. Don’t be fooled by their marketing. They want a return to the pre-scientific days when health gurus could sell any snake oil they want at exorbitant prices, with any hyped claims that they want, without going through all that tedious science.
They want to get rich at the expense of the public health and the health of our patients, and we are failing our patients and our professional duty unless we fight against CAM as vigorously as possible.

   The Bait and Switch of Unscientific Medicine
Steven Novella
Link
Savvy consumers are familiar with the classic scam of the “bait and switch” – in practice if not the term itself. My wife and I ran across it when we were shopping for our first car. We needed a bargain and so we were attracted to the ads that promised a new Colt for only $9,000 (that’s the bait). Of course when we got to the dealership they were all out of Colts with the configuration advertised, but they had plenty of others that had different options that cost several thousand dollars more (that’s the switch).
It’s a basic and very successful form of deception, and so even though there are laws against such practices it is impossible to eliminate in all its various and more subtle forms. It even permeates scientific, political, and other intellectual endeavors – anytime a more palatable idea or claim is put forward to represent the less acceptable truth.
Science, however, requires transparent honesty to function properly, and therefore scientific practitioners must vigilantly guard against the cognitive bait and switch. Generic intellectual virtues incorporate this vigilance – they include the need to unambiguously define terms, to make claims as specific and operational as possible, and the use of valid logic. Beware of any claims that subtly violate these rules because they are probably setting you up for a bait and switch.
The purveyors of unscientific medical claims have become as expert at this classic deception as the slickest used-car salesman – in fact they have left the hawkers of dubious transportation in the dust.
Practitioners of health fraud, unscientific and sectarian medical practices have learned how to apply the bait and switch at multiple levels – with respect to specific modalities and at higher categorical levels. First I will address some of the more common sectarian practices and show how they apply this deception, whether consciously or out of intellectual sloppiness.
Chiropractic
Chiropractic is perhaps the most common and egregious example of the bait and switch in medicine. The deception begins with the name itself – “chiropractic” fails the basic test of transparency because it is not unambiguously defined. There are in fact numerous professions doing very different things and employing mutually exclusive philosophies under the banner of “chiropractic.”
Therefore someone may go to see a chiropractor and think they will be seeing a medical professional who will treat their musculoskeletal symptoms, but in reality they will see the practitioner of a cult philosophy of energy healing. So-called “straight” chiropractors (who make up an estimated 30% of all chiropractors) still adhere to the original philosophy of chiropractic invented by “magnetic healer” D.D. Palmer, which is based upon the claim that an undetected life energy called “innate intelligence” flows through the spinal cord and nerves and is responsible for health. Such chiropractors will treat any disease or ailment with spinal manipulation.
Most other so called “mixer” chiropractors reject the notion of innate intelligence either partially or entirely, but still incorporate other pseudosciences into their practice. Chiropractors are the primary practitioners of homeopathy, applied kinesiology, and iridology in the US. The bait – claims that chiropractors are medical practitioners with expertise in the musculoskeletal system. The switch – practitioners of discredited pseudosciences that have nothing to do with the musculoskeletal system.
A more subtle form of the bait and switch among chiropractors is the treatment of musculoskeletal symptoms with standard physical therapy or sports medicine practices under the name of chiropractic manipulation. Ironically, the more honest and scientific practitioners among chiropractors are most likely to commit this subtle deception. The problem comes not from the treatment itself but the claim that such treatments are “chiropractic.”
Using techniques like massage, range of motion exercises, strength-building exercises, and mobilization of joints are all legitimate science-based techniques used by physical therapists and physicians with specialties in physiatry, orthopedics, and sports medicine. Some chiropractors also use similar techniques -and with good results. But by doing so and calling it “chiropractic” it legitimizes the pseudoscientific practices that are very common within the profession – like treating non-existent “subluxations” in order to free up the flow of innate intelligence.
Acupuncture
Many studies of acupuncture employ this latter form of bait and switch – studying something that is not really acupuncture but calling it acupuncture and thereby legitimizing pure pseudoscience. Acupuncture refers to a Traditional Chinese Medicine (TCM) pre-scientific philosophy of health and illness that is based (very similarly to straight chiropractic) on the flow of life energy called “chi”. Fine needles are inserted into special locations on the body called meridians for the purpose of improving, freeing, and balancing the flow chi.
No study has demonstrated that chi exists, or that acupuncture of the meridians has any specific effect. A look at the entirety of the acupuncture literature, in fact, clearly demonstrates that it does not matter where the needles are placed (or how deeply, or if they are manipulated in ways that are designed to manipulate chi). This means that any effects of sticking acupuncture needles into a patient are non-specific – they are not related to the flow of chi. There may be some small non-specific physiological effects – such as counter-irritation reducing pain or inhibiting nausea – but even these claims remain elusive and controversial.
But some studies of “acupuncture” show a larger and more reliable effect, usually on pain. What these studies have in common is that they employ “electrical acupuncture” (for example). Electrical acupuncture, however, is not acupuncture – it’s transdermal electrical stimulation, which is a scientific practice that has proven efficacy in the treatment of pain. Giving transdermal electrical stimulation through acupuncture needles, calling that “electrical acupuncture” and then using positive results to conclude that acupuncture works – is an elaborate bait and switch.
Homeopathy and Herbal Medicine
Homeopathy is another dubious practice based upon pre-scientific notions long discredited. A truly homeopathic remedy is diluted to the point that no active ingredients remain – so a homeopathic remedy is pure solvent, either water or alcohol, sometimes placed on a sugar pill (making a homeopathic pill a literal placebo). But much of the public believes that “homeopathic” simply means “natural” or “herbal”. Few realize that it means that the so-called medicine contains nothing.
But some homeopathic products are not even truly homeopathic – meaning they are not based upon homeopathic principles but rather contain measurable amounts of real drugs. This is now not uncommon for over-the-counters lotions and cold remedies. For example, this pimple cream contains many active ingredients at a “1x” dilution – that’s only 1:10, meaning there is still actual molecules of drugs in the cream. That’s not homeopathic.
A similar and more subtle deception is used in the marketing of herbs. They are sold and regulated as “natural supplements” when in fact they are drugs. Herbs contain chemicals that have pharmacological activity and they are usually taken in doses that are insignificant in terms of their nutritional content, but are significant in terms of their pharmacological activity. The bait is that they are nutritional supplements, the switch is that they are unpurified drugs.
CAM – the Biggest Bait and Switch
All of these modalities fall under an artificial category – so called “complementary and alternative” medicine (CAM) or “integrative” medicine that was created as the ultimate bait and switch.
The deception is largely two-fold. The first is to include modalities within this false category that are legitimate, like nutrition, exercise, physical therapy, relaxation, etc., – and then claim that these legitimize the entire category of CAM, even the far-out stuff like homeopathy. This is just a higher-order version of including physical therapy modalities within the umbrella of chiropractic, for example.
Essentially, at every level terms are used that are vague and poorly defined, and in practice are designed to be all-inclusive so that the purely pseudoscientific can ride on the coattails of the barely plausible or even the legitimate.
This is the opposite of what happens within scientific disciplines – where terminology evolves over time to become more discriminating, unambiguous, and specific. To take an example from my specialty, multiple sclerosis (MS) has been divided into sub-diagnoses: relapsing remitting and chronic progressive. Divisions multiplied and now there are seven subtypes – for example, primary progressive, and progressive remitting.
This increased “splitting” as it is called is not just a pedantic dedication to precision, but a recognition that such divisions may reflect differences in underlying mechanism of the disease. Also, different subtypes of MS respond differently to different treatments. Some drugs work in relapsing remitting but not in primary progressive. Part of the motivation for finer categories is to more accurately predict which MS patients are likely to respond to which treatments.
Contrast this to the maximally broad and imprecise but common marketing slogan “chiropractic works.”
The second kind of deception created by the category of CAM is in the language used itself – “complementary” and “integrative.” What, exactly, are CAM modalities integrating? On close examination it is quite clear – the movement is an effort to mix unscientific, disproved, and dubious modalities into scientific medicine. The bait is that CAM offers legitimate alternatives, the switch is that it primarily promotes treatments that don’t work or are at best untested and highly implausible.
If there were truth in marketing then we would have the Office of Implausible Medicine, the Journal of Bad Medical Science, the Center for Rejected Therapies, and the Institute of Dubious Medical Claims – all under the umbrella of unscientific medicine. It used to be called, even more simply, “health fraud.”
Conclusion
The public would be best served if they were given a reasonable expectation of honesty and clarity in the health marketplace. Instead, increasingly they are being given the old bait and switch, and increasingly the governments and institutions of medicine that should know better are instruments of this deception.
Science and intellectual honesty require that terms mean something specific and definable. Unscientific health modalities have become very savvy at using terminology to confuse rather than inform, to deceive rather than to illuminate.
It is the simplest form of the bait and switch – just call a product or service something that it isn’t.

Bias and Spin: Acupuncture and Chiropractic
Mark Crislip on March 6, 2015
Tweet

We all construct our narrative based on our biases and spin the facts so that the narrative confirms our biases. Among other characteristics, what separates an SBM provider from a SCAM provider is realizing that biases are always active and apply to me as well as everyone else.
My biases are simple: I am skeptical that humans can reliably understand reality without assistance and the best source of assistance is science.
I have a job where the expectation is that I will change practice as information changes. How I practiced medicine 30 years ago is very different from how I practice it today. Still, I note it is harder and harder to change my approaches as I get older. I get more set in my ways and it takes more effort to change as new studies are published. Sometimes it seems almost physically difficult.
Again, it is expected that not only will I change my mind over time, as the sole ID doctor at my hospitals, I will be the one to lead the change. Imagine how much harder it would be to change your mind if you were committed to a universal truth such as those that are alleged to underlie reiki or chiropractic. Years committed to a pseudo-medicine probably renders changing one’s mind virtually impossible.
I will admit there is one bias I cannot fathom, the conspiracy/pharmaceutical shill world view. The idea that those of us concerned with pseudo-medicine are doing so because we are paid by big pharma is just so weird. There are a number of articles on Pubmed concerning medical conspiracies, but the search term “pharmaceutical shill’ yields nothing on either Google scholar or Pubmed. It is a bias that evidently has no research to help understand why people have that opinion. I suspect it is derived in part from the often-remarkable sums paid to some doctors to pimp medications to other doctors. But I can’t even find how common the belief is.
I can kind of understand why people practice and/or participate in the various SCAMs given my understanding of the various fallacies and dysfunctional ways the mind works. It is easy to see how both the practitioners and patients misjudge the efficacy of pseudo-medicine. We have evolved to survive reality, certainly not to understand it.
Bias leads to spin, and it is curious how bias can warp the reports in the medical literature.
It is a curious fact that acupuncture studies from the East are virtually always positive:
Research conducted in certain countries was uniformly favorable to acupuncture; all trials originating in China, Japan, Hong Kong, and Taiwan were positive, as were 10 out of 11 of those published in Russia/USSR. In studies that examined interventions other than acupuncture, 405 of 1100 abstracts met the inclusion criteria. Of trials published in England, 75% gave the test treatment as superior to control….No trial published in China or Russia/USSR found a test treatment to be ineffective.
I find that fascinating. Is it publications bias i.e. only positive studies are published? Or is it, due to deep cultural bias combined with poor methodology (adequate blinding and placebo), that all studies are indeed positive, a massive example of N-rays in action?
Given that:
pronounced bias due to lack of patient blinding in complementary/alternative randomized clinical trials
leads to:
nonblinded patients exaggerated the effect size by an average of 0.56 standard deviation
and the effect is more pronounced in acupuncture trials, I suspect it is a combination, especially given a similar results with yoga studies out of India:
Positive conclusions were reached in 277 RCTs (91%); with more positive RCTs being conducted in India than elsewhere (odds ratio=24.8; 95% confidence interval=3.3, 184.5; p<0.001) while type of journal was not associated with the direction of the conclusions (odds ratio=1.2; 95% confidence interval=0.5, 2.9; p=0.828).
CONCLUSIONS:
RCTs on yoga that are conducted in India have about 25 times the odds of reaching positive conclusions as those conducted elsewhere.
Amazing how the positive results of large swaths of pseudo-medical studies can be biased by the culture in which they are done. It would appear that the country of origin may have a far more positive effect on a SCAM study outcome than the effect of when a real medication trial is funded by pharmaceutical companies. Money, as Cindy Lauper noted, changes everything, but world view with its concomitant bias is likely more important. Ioannidis was understating the issue when he noted:
Prejudice may not necessarily have financial roots. Scientists in a given field may be prejudiced purely because of their belief in a scientific theory or commitment to their own findings. Many otherwise seemingly independent, university-based studies may be conducted for no other reason than to give physicians and researchers qualifications for promotion or tenure. Such nonfinancial conflicts may also lead to distorted reported results and interpretations.
It is amazing how such bias can lead to the inability to see effects even when it is staring you in the face. My favorite example is “Decreased Risk of Stroke in Patients with Traumatic Brain Injury Receiving Acupuncture Treatment: A Population-Based Retrospective Cohort Study” where they conclude:
Patients with TBI receiving acupuncture treatment show decreased risk of stroke compared with those without acupuncture treatment.
even though patients:
with acupuncture treatment had higher proportions of using anticoagulants (2.3% vs. 1.7%, p = 0.0003), antiplatelet agents (28.7% vs. 22.3%, p<0.0001), and lipid-lowering agents (24.7% vs. 18.4%, p<0.0001) compared with those had no acupuncture treatment.
The use of anti-stoke medication was then ignored as the possible explanation of the ‘effects’ of acupuncture.
Of course, acupuncturists are not the only ones who cannot look at data and see the obvious.
Chiropractors like to point to “Risk of Vertebrobasilar Stroke and Chiropractic Care Results of a Population-Based Case-Control and Case-Crossover Study” as evidence that their particular pseudo-medicine does not cause strokes.
We found no evidence of excess risk of VBA stroke associated chiropractic care compared to primary care.
Although as I have mentioned in the past the paper does suggest that in a group that should not be having strokes, the young, seeing a chiropractor is associated with having a stroke.
Table 3 has the smoking gun. The highest odds ratio (14) for a stroke is 12, 4 times any other, in the age less than 45 in the first 24 hours after a visit to a chiropractor.
Saying that a paper does not support chiropractic as a risk of stroke when the data suggests it probably does continues. In the study “Risk of Stroke After Chiropractic Spinal Manipulation in Medicare B Beneficiaries Aged 66 to 99 Years With Neck Pain” by a chiropractor, suggests the older you are, there may be greater the risk of stoke after visiting a chiropractor.
They conclude that
Among Medicare B beneficiaries aged 66 to 99 years with neck pain, incidence of vertebrobasilar stroke was extremely low. Small differences in risk between patients who saw a chiropractor and those who saw a primary care physician are probably not clinically significant.
Given that the chiropractic cohort was both younger and healthier, and the rarity of stroke, I am not sanguine that this study can be used as evidence that chiropractic neck manipulation is safe and it suggests that, as the older you get, the more likely chiropractic will be associated with a late stroke, a 10% increase at 30 days that was statistically significant. But if due to chiropractic cannot be determined from the study
They suggest it may not be clinically important. While a 10% change in a pain scale may not be clinically relevant, I am not so sure a 10% increase in stroke has the same degree of triviality.
I am also not sanguine the study supplies evidence that chiropractic is safe; it is more spin than fact. Like the Cassidy study, it suggests that a risk of stroke is lurking in the background in some populations, is rare, and hard to see. For now I will stick with the preponderance of data, nicely summarized by the AHA position paper:
Although current biomechanical evidence is insufficient to establish the claim that CMT causes CD, clinical reports suggest that mechanical forces play a role in a considerable number of CDs and most population controlled studies have found an association between CMT and VAD stroke in young patients. Although the incidence of CMT-associated CD in patients who have previously received CMT is not well established, and probably low, practitioners should strongly consider the possibility of CD as a presenting symptom, and patients should be informed of the statistical association between CD and CMT prior to undergoing manipulation of the cervical spine.
And when I see an article called “Is there a role for neck manipulation in elderly falls prevention? – An overview“, I think NONONONONONO.
That’s my bias, and I am sticking to it.

Motivations
Mark Crislip on December 27, 2013
Link
We received the following letter:
Your blog, the SBM page, has come up for me several times in my research. I’m an RN trying to research cancer treatment, for myself, I am the patient. I’m also a licensed massage practitioner with a 30 year history using “alternative” or “complementary” medicine successfully to treat myself for various things. When your blog has come up I’ve read into it, picking up some useful information.
My question is, why the sarcasm? Why do you and the responders on your site have such disdain and anger? Why do you feel threatened? Your physician status gives you the top of the totem pole, the extensive training, the authority and the privilege. Very doubtful you are going to be knocked off the top of that totem pole in this or any several more lifetimes to come.
I say that I’ve picked up a few useful tidbits from your site, but in the main I am troubled by the anger and disdain that is expressed toward practitioners who obviously are helping other human beings. Patients who have to pay cash for a service because their insurance does not cover it do not keep going back to a practitioner who does not help them. What are you afraid of?
I am served well by a naturopathic oncologist who ordered me some homeopathic cream for my breast surgery, as well as a pharmaceutical grade supplement of multiple vitamins. My breast suffered a hematoma post op, and it turned black. The opposite breast also turned black from blood that drained to it as I lay on my unaffected side. In 24 hours I had one and one-half black breasts. The post op nurse and my PCP both stated it was the worst bruising they had ever seen in their careers, and neither of them are less than 40 years of age.
With the help of the cream and supplements, the bruising is resolving in days. Not weeks. Days. The post op nurse told me to expect this dark bruise to linger for weeks or months. I credit the cream and the supplements, neither of which the surgeon ordered. A top surgeon at a top university hospital.
Something is wrong with the premise or the presentation on your site. If physicians have this much disdain for others who are obviously helping their patients, then something needs to be dialed in, dialed back, or the baby thrown out with the bath water.
First, do no harm.
Since the holidays tend to fill my alleged free time with family commitments and I need to rewrite some RSS feeds (am I lucky or what?), I thought I would take this opportunity to answer the letter since it will require zero research. The other contributors on the team probably have their own motivations for participating in this blog and in science-based medicine in general. Perhaps they will weigh in in the comments. This is why I spend my time working away in the endless field of SCAM.
First, I like to write. I enjoy the process of trying to explain a complex topic but also like to be clever or humorous in the process. I like to make me laugh and, I hope, others. It is an enjoyable intellectual process. This brings up issues of tone in the blog, and there will be disagreements amongst the participants in the blog as to tone. I look at the blog as tending towards the informal and the personal. The entries should reflect the author and the few blogs I read regularly are less due to the content and more due to the personality behind the blog. I want style AND content and the former is rare. Many people can write well organized and reasoned essays; few can do it with personality. So I try to have my entries reflect something of who I am and express my personal reaction to the topics covered as honestly as I can.
It can, and should, come across as disdainful and sarcastic, for reasons that will become clear later in the essay. Disdain not for the people involved, I hope, but the ideas and concepts of SCAM. It is the sin, not the sinner, that I hope is the target of some of my loathing.
I like to learn, and there no is better process to help one learn than to try and explain a topic someone else. It helps in my infectious disease world and it helps as well in the world of SCAM.
I am also endlessly fascinated in the almost-too-numerous-to-count ways humans think and think badly. The variations of human cognitive pathology represented by SCAM proponents are amazing. The SCAM world has endless examples of how to reason poorly and the inability for people to recognize that they are thinking poorly. I remain amazed that humans are consistently horrible at understanding the natural world without the support of the scientific method.
I have become a better physician as a result of understanding all the logical fallacies to which humans are prone and applying them to myself. That, I would suspect, is the main issue that separates the SCAM world from the skeptical world. Those who participate in pseudo-medicine are generally unable to apply the concepts of critical thinking to themselves. I suspect that to do so is an unnatural act that most cannot, or will not, do.
But the main reason I spend my time writing and podcasting on pseudo-medicine derives from why I originally became involved in the SCAM world.
It was in the early 1990’s, my first year or two in practice, when I received a consult for a case of gangrene, not a typical cause of infection in the industrialized West.
The patient was a female in her mid-twenties. Sometime earlier she had been diagnosed with a tumor of her leg; I remember it as an osteosarcoma but that detail could be mistaken. It was some sort of musculoskeletal tumor. Rather than an amputation and chemotherapy, which could have been curative, she opted for the care of a naturopath who treated her cancer with alkaline therapy, since, as the naturopath told her, cancer is due to acid.
The cancer progressed, grew and parts of it outgrew the blood supply, leading to dead, infected tissue. Her naturopath told her this was her body rejecting the tumor. By the time I saw her, most of her leg was black, ulcerated, dead and with a stink that I hope you never experience. There are few smells as foul as rotting human flesh.
The only reason she was in the hospital was that she had fainted from anemia and continued to refuse any medical or surgical interventions for her leg, which she told me was going to be cured. She believed in the pseudo-medicine she was receiving. She bet her life on it. That night the tumor eroded into a major artery and she bled to death in her sleep.
One of the curious phenomena is the demonization of those of us in science-based medicine: we are big pharma shills or insurance company shills or we are motivated by fear of our position being removed. Or we are angry. Yeah. I will cop to that last one. Anger. At people spending their time, their hope, their money, their health and their lives on worthless nonsense.
Reality, as I have mentioned in the past, is a honey badger. It don’t care what you think, if you misstep or are unlucky, reality will hurt or kill you. Life is short. Health is rare.
There is a reality that proponents of pseudo-medicine deny but we are all subject to its rules. You may think you can harness the energies of the universe to make yourself invulnerable to harm. The sword don’t care, it still cuts. That martial artist is the archetype of all believers in un-reality. We have not evolved to understand or comprehend reality. For understanding we need science.
The SCAMs about which we write on this blog and elsewhere are based on fantasy and the endless ability for people to fool themselves and others. And if you spend your time treating illness with fantasy you could end up bleeding to death in the middle of the night from what could have been a curable cancer.
Yes. I know. Medicine kills. But less so every day. Since 2006 in my hospital system we estimate that we have prevented over 2 000 infections and 200 deaths. How? By applying reality-, science-, and evidence-based medicine to the treatments of our patients. As Chair of Infection Control for my hospitals I have seen the enormous work we have done to improve care and make medicine safer. None of the infections we have prevented or the deaths we have avoided have been in any way due to any SCAM.
As far as I know there has yet to be any quality improvement in any SCAM except for the use of disposable acupuncture needles. Then just watch a video of acupuncture and see if you can see anything that resembles good infection-control techniques. A key feature of reality/science-based medicine is that it reacts to information and changes slowly, painfully, erratically, for the better. What we do works. Sometimes. And when it doesn’t it is abandoned or improved.
I am a doctor. As I mentioned in another entry, I see medicine as a calling. I, and many of my colleagues, have literal responsibility for the health, life and wealth of people. We see people every day who lose all three because of bad decisions or a lack of insurance.
I am always more than a little saddened by the intravenous drug users I see. There are those whose main attitude is that IVDUs made a choice and get what they deserve. I see a life wasted and it is sad. Or the AIDS patient who does not come in due to a lack of insurance until the immune system is beyond salvage. Or the middle-aged black female whose first manifestation of high blood pressure is a large stoke. Every day can be a reminder of how fragile and transient life is.
I am not worried about me and medicine. I am not worried about my position, I worry about the next person who may needlessly die or be injured by a treatable illness because they opted instead to treat their cancer with alkaline or deny that HIV causes AIDS or treats their diabetic foot infection with honey.
So when people throw away their life, their health and their wealth pursuing alternative interventions that do not work, that cannot work, it tends to be a motivator. Most SCAM is based on fantasy and fantasy kills and maims and bankrupts. And that makes me angry. And so I continue to always, scribble, scribble, scribble.
Often we get the phase ‘do not throw the baby out with the bathwater’. In the world of SCAM, I am not certain what is the baby (the result?) or the bathwater (the SCAM therapy?). It is a non sequitur. There is no pseudo-medical bathwater that will clean any baby. A more apt cliché concerns emperors and his new clothes.
And as to the idea of ‘first, do no harm’, it is a concept that has never been true in medicine. All actions, and inactions, can have consequences that can lead to harm. There is no way to avoid harm. Glib aphorisms may be a nice way to avoid considerations of complex phenomena, but, like much of SCAM, have little to do with reality.
My job, my responsibility, my calling as a physician is, to the best of my ability, diagnose and treat your illness. That cannot be done by applying ideas that are based on fantasy to problems that come from reality.
I do not doubt that most of those practicing pseudo-medicine have nothing but good intentions. That does not make what they do good, correct, or effective. This blog and its cousins exist because we care that people suffer and die and lose money and hope and time by participating in nonsense. It is a waste of lives that will never have a surfeit of the characteristics that pseudo-medicine steals from the people who use them. As Terry Pratchett noted:
The road to hell is paved with good intentions. (This is not actually true. The road to Hell is paved with frozen door-to-door salesman. On weekends, many of the younger demons go ice skating down it.)
Others have summarized my position far more elegantly than I am able:
No man is an island, entire of itself; every man is a piece of the continent, a part of the main. If a clod be washed away by the sea, Europe is the less, as well as if a promontory were, as well as if a manor of thy friend’s or of thine own were: any man’s death diminishes me, because I am involved in mankind, and therefore never send to know for whom the bells tolls; it tolls for thee.

The false dilemma of David Katz: Abandon patients or abandon science
David Gorski on May 5, 2014
Link
Dr. David L. Katz is apparently unhappy with me. You remember Dr. Katz, don’t you? If you don’t, I’ll remind you momentarily. If you do, you won’t be surprised. Let me explain a bit first how Dr. Katz recently became aware of me again.
Last week, I posted a short (for me) piece about something that disturbed both Steve Novella and myself, namely Traditional Chinese herbalism at the Cleveland Clinic? What happened to science-based medicine? Steve had blogged about it as well a couple of days earlier. In actuality, it was a post that had originally appeared at my not-so-super-secret other blog, and, in my characteristically slightly arrogant way, I thought it was good enough that it deserved to be showcased here at Science-Based Medicine. To my surprise, Maithri Vengala over at The Healthcare Blog noticed and asked me if I would mind letting her post it over there. Never being one to turn down a request to showcase my work to a wider (or at least different) audience, I gave her my permission. The result was that my post ended up being published here, and I thought nothing more of it.
Until yesterday, that is.
Yesterday, thanks to the magic of Google Alerts, I became aware that Dr. David Katz was very unhappy with my post. At the very least, he strongly disagreed with it, so much so that he felt the need to respond. Naturally, he chose as his venue The Huffington Post, which is well known as a bastion of quackery, antivaccine pseudoscience, and Deepak Chopra-inspired magical thinking, to respond. Indeed, so bad is HuffPo (as it’s “nicknamed”) that Steve Novella and I have both referred to it as waging a “war on medical science,” and HuffPo has been a frequent topic of discussion on this very blog for its abysmal record of publishing pseudoscience, a record that goes back to its very beginning in 2005, when antivaccinationists flocked to the fledgling blog and news site. And that doesn’t even count all the nonsense from Deepak Chopra and even promotion of outright cancer quackery.
The title of Dr. Katz’s response is very telling: A Holistic View of Evidence-Based Medicine: of Horse, Cart and Whip. Yes, it’s a rather odd metaphor. Dr. Katz’s apparent viewpoint is that that integrative medicine is the bees’ knees, the “best of both worlds,” and that stodgy old physicians like myself who insist on science are just being closed minded. Wait, that’s actually not quite what he said. I’m sorry, but I’m thinking of the old David Katz, whom I’ll describe in a moment. The new, improved David Katz is completely about the evidence, so much so that he trumpets his credentials thusly at the beginning of his article:
First, I am a card-carrying member (well, I would be if they issued cards) of the evidence-based medicine club. I am a conventionally trained Internist, and run a federally funded clinical research laboratory. I have taught biostatistics, evidence-based medicine, and clinical epidemiology to Yale medical students over a span of nearly a decade. I have authored a textbook on evidence-based medicine.
But on the other hand, I practice Integrative Medicine, and have done so for nearly 15 years. And I represent Yale on the steering committee of the Consortium of Academic Health Centers for Integrative Medicine.
That’s nice.
It is rather telling that Dr. Katz apparently seems to feel compelled to begin his rebuttal with, in essence, an appeal to authority—his. Look, he seems to be saying, I’m not one of those airy-fairy alternative medicine hippies like Andrew Weil who believes that natural is better. I’m a real scientist! I have federal funding! I’ve even taught biostatistics and written a book on evidence-based medicine (13 years ago). Respect me, dammit! Well, it’s true that Dr. Katz has many more publications than most of us do, and he does indeed run an federally-funded center, while I haven’t had NIH funding since 2011, thanks to the funding crunch at the NIH leading to increased competition that I haven’t (yet) been able to crack, but so what? It is the quality of evidence and argument that matters. That’s why I rarely dwell long (if I mention them much at all) on my own qualifications, even when I’m writing about topics that fall under the purview of my specialty that I might reasonable be considered to be an expert in, such as screening mammography. It’s just…unseemly.
There’s also the not-insignificant issue of how Dr. Katz basically begins his response by attacking what can only be described as a straw man whose size rivals that of one that might be seen at a Burning Man gathering. It apparently irks him mightily that I described one of the flaws in reasoning that herbalists frequently use to justify the use of herbal medicine over pharmaceutical medicine as believing that somehow “natural” is better, that the ingredients in herbs are “synergistic,” even though the evidence supporting such an assertion is incredibly unpersuasive, with less than a handful of known exceptions. Dr. Katz seems to think that that statement was aimed at him specifically. So vociferous is the umbrage that he takes at the suggestion that he objects to it multiple times in just his one post, beginning with this:
I did not go into Integrative Medicine because I believe “natural” is reliably better or safer than “scientific.” I respect the often considerable prowess of modern medical technology and pharmaceuticals. And, frankly, I have never much cared whether a therapy derived from a tree leaf, or a test tube. I have cared about whether it was safe, and whether it was effective.
Well, no one ever said that Dr. Katz went into integrative medicine for that reason, and I note that, in the context of my post, that criticism was leveled more at herbalists and supporters of using herbal medicine in CAM than anyone else. Yet, Dr. Katz seems to take the criticism very, very personally. It reminds me of the scene from The Godfather, when Michael Corleone first proposes killing Sollozzo and Captain McClusky (Sollozzo ordered a hit on Michael’s father Vito Corleone, who unexpectedly survived the attempt on his life, and McClusky beat Michael up when he foiled a second assassination attempt by Sollozzo’s men on Vito as he was in the hospital recovering from the serious wounds from the first assassination attempt), at which Michael’s brother Sonny laughs and tells him, “You’re taking this very personal. Tom, this is business, and this man is taking it very, very personal.” The only reason I can think of that Dr. Katz would react so self-righteously to criticism of herbal medicine is because he taking my criticism as personal as Michael Corleone did Sollozzo’s attempts to kill his father. He does, after all, apparently use herbal medicine while viewing himself as a Real Scientist and his pursuit of “integrative medicine” as being truly evidence-based. Unfortunately, he appears to be deluding himself, and his protestations carry a not-so-faint air of whistling past the graveyard.
Before I go on, I will, however, point out that I actually do agree with Dr. Katz on one point, and I’m sure that Steve and pretty much everyone else here at the SBM Blog also agree. We don’t care whether a treatment is “natural.” Nor do we much care whether a therapy is derived from a tree leaf or a test tube. Like Dr. Katz, we only care about whether it works and is safe. Again, the implication on Dr. Katz’s part by emphasizing that seems to be that somehow those of us who criticize “integrative medicine” care more about where a treatment came from than the evidence base behind it. We don’t. It was not from people like Dr. Katz that we got the maxim, oft-repeated in various forms by skeptics as diverse as Richard Dawkins and Tim Minchin, “There is no such thing as alternative medicine. There is medicine that has been scientifically proven to work and medicine that hasn’t. What do you call alternative medicine that’s been scientifically proven to work? Medicine.”
Dr. Katz also tries to convince his readers, using a typical favorite trope of supporters of “complementary and alternative medicine” (CAM) and integrative medicine, that evidence-based medicine (EBM) is not particularly evidence-based. I’ll give him credit that at least he didn’t cite the usual myth that only 15% of current medical practices are “evidence-based,” although he did cite the high end of these tropes, namely that only 50% of medical practices are evidence-based. As Bob Imrie discovered when he tried to track down the origin of that claim, the 15% figure comes from a small study in the north of England from 1961 that was looking at insurance reimbursement. Other, somewhat higher estimates (10-35%) appear to come from a chapter by Kerr L White entitled “Archie Cochrane’s legacy: an American perspective” in the book Non-random Reflections on Health Services Research: on the 25th anniversary of Archie Cochrane’s Effectiveness and Efficiency. In actuality, most estimates these days are considerably higher. Of course, it varies by specialty, but existing evidence suggests that around 76% of interventions are supported by at least some form of compelling evidence, be it randomized controlled trials (RCTs) or strong observational evidence. Indeed, as far back as 1995 we knew that for inpatient medicine, at least, over 50% of interventions were based on RCTs and that 82% of interventions were evidence-based.
Dr. Katz’s defense of integrative medicine, spurred by the criticism of the herbal medicine clinic over at the Cleveland Clinic, boils down to three points:
     1.  Evidence is not a reliable differentiator of conventional and alternative medicine.
     2.  To the extent that evidence does differentiate conventional and alternative medicine, it’s often because — in the pursuit of evidence — cart and horse routinely swap positions and money cracks the whip.
     3.  Evidence is not black or white. It comes in shades of gray.
In other words, Katz uses arguments that consist, basically, of a tu quoque argument (your evidence based medicine has a lousy evidence base, too!); special pleading based on the claim that no one wants to fund studies of “integrative medicine” because no one can make money off of it; and one that argues against a straw man, namely the insinuation that defenders of SBM assume that evidence is cut and dried, black or white, yes or no, a claim that I’ve yet to see any of us make. (Maybe he is referring to the use of basic science principles to argue against homeopathy, which is about as close as we get to such proclamations.)
I’ve already mostly dealt with #1 by showing that the evidence base for EBM is far higher than Dr. Katz would like you to believe. I concede, it should be higher, but compared to anything in “integrative” medicine, it’s not even close, Dr. Katz’s citation of a paper of his from 11 years ago about “evidence mapping” notwithstanding. One notes that evidence mapping, as described in the paper by Dr. Katz and colleagues, doesn’t really show what he says it shows in that it doesn’t really demonstrate that there is high quality evidence for CAM that is equivalent (or even close to equivalent) to that which exists for EBM. In reality, the paper simply describes a method that might or might not have some validity. That’s all.
The second objection is nothing more than the oft-repeated whine from supporters of CAM and integrative medicine that their methods aren’t studied, that they’re given short shrift, because big pharma can’t make money off of them. A frequent variant of this particular line of special pleading is to impugn big pharma as being so hopelessly corrupt that one has to consider CAM and integrative medicine as potentially viable alternatives, often with a not-so-subtle insinuation that big pharma isn’t interested because there’s no money or even that big pharma is actively “suppressing” alternatives, the better to protect its profits. Whenever I see such an argument, I like to paraphrase Ben Goldacre’s adage that says something like:
@madeforshade just because there are flaws in aircraft design that doesnt mean flying carpets exist
— ben goldacre (@bengoldacre) January 30, 2013
Or:
Quacks citing problems in pharma make me laugh. FLAWS IN AIRCRAFT DESIGN DO NOT PROVE THE EXISTENCE OF MAGIC CARPETS.
— ben goldacre (@bengoldacre) January 31, 2013
Or:
Problems in medicine do not mean that homeopathic sugar pills work; just because there are problems with aircraft design, that doesn’t mean that magic carpets really fly.
In other words, just because there are problems with drug development and how big pharma handles drug trials does not mean that the pseudoscientific methods of CAM and integrative medicine “work.” Just because, as Dr. Katz demonstrates, a purified drug (coenzyme Q10) might be effective in preventing death from congestive heart failure and that the clinical trials that failed to find a benefit a decade ago were small, that doesn’t mean “integrative medicine” is valid, nor does it mean that herbalism works. Such claims are non sequiturs. In fact, I even challenge Dr. Katz as to why coenzyme Q10 should even be called “alternative” or “integrative” at all. If it works, it’s science-based medicine. To turn Dr. Katz’s words back on him, it doesn’t matter where it came from or if it’s natural. I would also argue that, since it is a purified chemical and that is taken like a drug, coenzyme Q10 as used for treating congestive heart failure should be considered a drug, just like beta blockers.
In any case, the answer to pharma is not to conclude that integrative medicine works, but to work on fixing the issues with pharma, publication of clinical trials, and funding of research, as Ben Goldacre is doing his part to do. In other words, if you agree that there is a problem with the distribution of research funds in this country, you don’t have to conclude that research into CAM remedies is being “suppressed” or ignored because pharma can’t make money off of them. The heart of Dr. Katz’s tortured metaphor seems to be that money is the “whip” that drives what is studied, leading to the reversal of cart and horse—or something. It is a rather incoherent metaphor. Dr. Katz’s odd metaphor aside, though, I’d say it’s quite the opposite if you look at it as a ratio of funding to promise (if it were possible to quantify promise). Given how wildly implausible many CAM treatments are (more on that later), the amount of money spent on them by NCCAM and other sources far outstrips any reasonable financial estimate of what positive research findings would be worth. Indeed, studying such remedies without compelling preclinical evidence really is putting the proverbial cart before the horse. Whether ideology (in the form of increasing acceptance of magical CAM thinking) or money from the National Center for Complementary and Alternative Medicine or Bravewell is the “whip” matters little. The cart has been put before the horse.
Dr. Katz’s third argument is a massive straw man, too:
Clinical decisions are easy if a treatment is known to be dangerous and ineffective, or known to be safe and uniquely effective. But what if a given patient has tried all the remedies best supported by randomized clinical trials, but has “stubbornly” refused to behave as the textbooks advise and failed to get better? Or what if a patient just can’t tolerate the treatments with the most underlying evidence? One option is to tell such a patient: See ya! But I think that is an abdication of the oaths we physicians took. When the going gets tough, we are most obligated to take our patients by the hand, not wave goodbye.
See what I mean? No one—and I mean no one—here argues for abandoning the patient who fails to get well when treated with science- and evidence-based treatments. Dr. Katz’s thinking is indeed muddied, as Kimball Atwood described so long ago. Indeed, if anyone is exhibiting black and white thinking here, it’s him. To him, if “conventional” science- and evidence-based medicine can’t help a patient, the choice is between abandoning the patient and embracing quackery. That’s a false dilemma in which Dr. Katz represents embracing quackery as the only other choice besides leaving the patient to his or her own devices. In fact, Dr. Katz even makes it more explicit here:
Integrative Medicine should not involve a choice between responsible use of evidence and responsiveness to the needs of all patients. It should combine the two. We should do the best we can with the evidence we have, but recognize that high quality evidence may start to dwindle before our patient’s symptoms start to resolve. We should resolve to confront this challenge with our patients, not leave them to fend for themselves.
Again, no one—and I mean no one—argues that EBM or SBM means choosing between the responsible use of evidence and responsiveness to the needs of patients. That is merely a convenient false dichotomy that Dr. Katz uses to justify embracing quackery to himself and to paint supporters of SBM as heartless bastards who care more about evidence than actual patients. Dr. Atwood had a good answer for such arguments, namely that a major role of the physician is as expert consultant and that morally we are obligated to provide recommendations that are rooted in evidence and science, not magic and superstition. In other words, we are obligated to tell the truth about the science behind the interventions we propose. Moreover, there are virtually no situations in which CAM or “integrative medicine” is the only option left, but Dr. Katz quite consciously portrays the choice thusly, as black or white, as CAM being the “only” choice after the failure of EBM for a patient. It is a gambit he has used many times before. Indeed, he portrays himself as having been “led to” integrative medicine” by this dilemma, while portraying himself as totally dedicated to science, an odd proclamation, given some of his previous statements.
Perhaps the most famous of these occurred a few years ago, resulting in a quote that caught a lot of people’s attention, including mine:
I think we have to look beyond the results of RCTs in order to address patient needs today, and to do that I’ve arrived at the concept of a more fluid form of evidence than many of us have imbibed from our medical educations…[Referring to a patient anecdote, Dr. Katz went on.] Now, we don’t want you on narcotics anymore than you want to be on narcotics. We initiated a course of acupuncture and over the next two to three months weaned him off narcotics. He was pain-free on acupuncture and subsequently transitioned into homeopathy. Now, I don’t care to get into a discussion of how or even whether homeopathy even works, but this guy had tried everything.
[…]
And the anecdotal evidence that homeopathy might be effective was brought up by the naturopaths. We tried it. It worked.
Of course, Dr. Katz neglected to describe exactly how he knew his intervention of acupuncture and homeopathy had “worked,” namely anecdotal evidence which, as we have seen many times in this blog, is incredibly prone to misinterpretation through regression to the mean and confusing correlation with causation, thanks to a number of cognitive quirks we humans all share. Indeed, science itself, not just science-based medicine is designed to minimize the effects of these quirks. In response to criticism, Dr. Katz went even further:
The view I expressed, and that guides our practice, is that human need goes on long after the results of randomized clinical trials start to run thin. I do not think doctor and patient should part company there. I believe that responsible use of the science we have should not preclude responsiveness to the needs of patients that fail to respond as textbooks say they should to that all-too-limited science. I believe, in other words, that patient need, not trial results, should be the ultimate master medical care must serve.
I learned devotion to evidence-based practice from my teachers and professional colleagues; I was pushed toward integrative medicine by the needs of my patients.
This was from 2008. Dr. Katz has been using the same false dichotomy for nearly six years now, minimum, and certainly much longer. It’s also very much a postmodernist view, in which Dr. Katz views science as nothing more than another “narrative” that he can abandon when he perceives his patient’s need driving him to do so. These are subtle distortions of language that are very powerful and incredibly useful for CAM advocates. RCTs, “what we know,” and science are all conflated as though they mean the same thing, leading to a false dichotomy in which the only choice when the edges of what RCTs tell us are reached is to rely on anecdotes, testimonials, and other forms of unreliable and subjective “evidence” in order to justify choosing unscientific treatment modalities, even abject pseudoscience like homeopathy. It’s the dilemma at the heart of alternative medicine (and, make no mistake, “integrative medicine” is nothing more than “integrating” alternative medicine into science-based medicine).
Particularly contradictory is this assertion:
The array of potential options extends, of course, to herbal remedies and nutriceuticals as well — the apparent focus at the Cleveland Clinic. And, more controversially, it potentially extends to modalities that conventionally trained clinicians find implausible, such as homeopathy or energy therapies. I won’t get too deep into such weeds today, but have done so before.
The paper referenced is an amazing bit of woo. In it, Dr. Katz tries to argue that the implausibility argument (i.e., when we argue that modalities like energy medicine and homeopathy are implausible to the point of being, for all practical purposes, impossible on basic science grounds alone because they violate multiple well-established laws of physics and chemistry and the “energy” in energy medicine has never been detected) is not so persuasive. He begins by describing organisms as being made up of molecules, which are made up of atoms, which are made up of subatomic particles, and how all matter is mostly empty space, so that when two people shake hands they are not actually “touching” each other, although they perceive it that way. We are thus “electromagnetic.” (Sounds like quantum to me.) Using language and flourishes that wouldn’t be out of place in Neil deGrasse Tyson’s update of Cosmos (in fact, I recall a segment recently that said essentially the same thing about two human beings touching each other), Dr. Katz marvels at this, particularly how amazingly implausible it seems to him, and then turns science on its head to make a massive argument from that very wonder and seeming implausibility. After describing how “inexplicable” he finds our existence and consciousness to be, Dr. Katz then writes:
I would not deny the implausibility of the therapeutic influence from not touching, nor contest the improbability of healing messages left behind by molecules diluted out of solution. But once we acknowledge that little could be more unlikely or wondrously implausible than a handshake, the topic of plausibility must be broached with greater purpose. I don’t understand homeopathy, or believe in it per se. I find it inexplicable and farfetched, but perhaps slightly less so than the experience of a caress, and incalculably less so than our existence.
We are. Therefore, I think, we must be plausible. But frankly, I don’t see how.
That’s right. We exist and our existence is implausible to Dr. Katz; so homeopathy and energy medicine, as implausible as they are on the basis of science, maybe aren’t so implausible after all, at least not to Dr. Katz. And just when you think it can’t get more flaky than that, Dr. Katz drives the point home:
We live out our lives within the bounds of perception that are but a thin layer of what is. Well beneath it, where reality meets bedrock, solar systems, software, and sushi are the same. This goes well beyond lifting the lid from Pandora’s box; it blows open the walls of Descartes’ carton. It is an admission that you and I, the lid and the box, poet and page, myth and math, Pandora and Descartes are the same stuff, and there is nothing much to any of us.
I am writing, you are reading, and we are thinking—and therefore we ostensibly are. But that we are is a veritable assault on plausibility. What we think we are is merely what we perceive ourselves to be.
Therefore, homeopathy and energy medicine aren’t implausible, you nasty skeptics! Q.E.D. Oh, and we are all made of star-stuff, too; so homeopathy and energy medicine must work. That really is the heart of Dr. Katz’s argument against rejecting scientifically implausible/impossible contentions. It’s just that bad.
In the end, Dr. Katz sounds very frustrated and unhappy—and, yes, defensive. At one point in his post he complains that it was “painfully clear” to him that he “could not make everyone better” and “modern medicine couldn’t make everyone better.” And it’s true. Most of the conditions that internists routinely take care of are not curable, but that’s not a reason to embrace magical thinking, as Dr. Katz does. I also confess that it’s the reason I went into surgery rather than internal medicine. After originally planning to go into internal medicine, in medical school I found that, personality-wise, I wasn’t suited to managing chronic diseases with no cure, like diabetes, hypertension, heart disease, and the like, although I greatly admire the primary care docs who can do this well. I couldn’t; so I went into a specialty where cutting out a tumor is usually curative. In other words, in collaboration with medical and radiation oncologists, I am in the fortunate position of being able to make the vast majority of patients I see better. Personally, I wonder if Katz would have been happier and less attracted to quackademic medicine if he had chosen another specialty, one not so involved in taking care of chronic degenerative diseases as internal medicine.
Maybe he wouldn’t have come to the conclusion that the only way he can help patients whom he can’t cure is to abandon science in the guise of claiming to be scientific. It’s a false dilemma that drives too many physicians to abandon their critical thinking skills as well.
ADDENDUM: Steve Novella has also dissected Dr. Katz’s article.

Integrative medicine, naturopathy, and David Katz’s “more fluid concept of evidence”
David Gorski on April 6, 2015
Link

Dr. David Katz is undoubtedly a heavy hitter in the brave new world of “integrative medicine,” a specialty that seeks to “integrate” pseudoscience with science, nonsense, with sense, and quackery with real medicine. In fairness, that’s not the way physicians like Dr. Katz see it. Rather, they see it as “integrating” the “best of both worlds” to the benefit of patients. However, as we’ve documented extensively here, on our personal blogs, and even in the biomedical literature (plug, plug), what “integrative” medicine means in practice is indeed what I characterized, the infiltration of woo into medicine. This infiltration seems to have started mainly in academia—hence the term “quackademic medicine” and “quackademia”—with the steady infiltration of nonsense into medical schools and academic medical centers, but has since metastasized to the world of community hospitals. This “integration” (or, as I like to refer to it, “infiltration”) has become so pronounced that a few years ago The Atlantic published an article entitled “The Triumph of New Age Medicine“, and just last December the Journal of the National Cancer Institute (JNCI) published a monograph full of articles touting “integrative oncology,” including guidelines recommended by the Society of Integrative Oncology (SIO) for the “integrative” treatment of breast cancer symptoms.
I mention Dr. Katz for two reasons. First, he’s taken another broadside at us at Science-Based Medicine in blog entry at The Huffington Post—where else?—entitled “Holism, Holes and Poles” that I’ve been meaning to address for a while. But before I address Dr. Katz’s most recent complaint against science-based medicine (SBM), it’s necessary to step back and look at some history.
A more “fluid” concept of evidence
Longtime readers might recall that Dr. Katz is on faculty at Steve Novella’s institution, Yale. More importantly he is (in)famous for making a statement that utterly blew me away seven years ago when I first encountered it, courtesy of David Colquhoun, portrayed in this video, specifically beginning at around 6:05:
Unfortunately, the original video I cited in my original post was taken down due to a copyright claim from Yale, but this one remains. The problem is that it is missing bits that were discussed when the talk was widely discussed in the medical blogosphere. However, the money quotes are still there, such as:
I think we have to move beyond the results of RCTs in order to address patient needs, and to do that I’ve arrived at the use of a more fluid concept of evidence than many of us have imbibed from our medical educations…[Referring to an anecdote of a patient with chronic pain Dr. Katz went on.]…Now, we don’t want you on narcotics any more than you want to be on narcotics. We initiated a course of acupuncture and over the next two to three months weaned him off narcotics. He was pain-free on acupuncture and subsequently transitioned into homeopathy. Now, I don’t care to get into a discussion of how or even whether homeopathy even works, but this guy had tried everything.
[…]
And the anecdotal evidence that homeopathy might be effective was brought up by the naturopaths. We tried it. It worked.
Later in the talk, he mentions an NIH consensus statement about the use of multivitamins that concluded that there was insufficient evidence to determine if they work and says:
And so I would argue that here, too, is an invitation to think more fluidly about evidence. Absence of evidence is not evidence of absence.
Whether or not absence of evidence is evidence of absence depends, of course, on how hard scientists have actually looked for evidence. Of course, the proper retort to such seemingly-sensible nonsense is to suggest that maybe—just maybe—you shouldn’t be making such confident recommendations until there is actually some convincing supportive evidence. Moreover, in the context of discussing a bunch of negative clinical trials that Dr. Katz apparently doesn’t believe, it is fairly clear that he’s implying that negative clinical trials are an “absence of evidence.” They are not. They are evidence. While from a frequentist perspective, it is true that a negative clinical trial does not necessarily mean that a treatment does not work (it just means that the results observed in the treatment and control groups were statistically indistinguishable, to put it in simple form), when negative clinical trials, particularly if they are well-designed, pile up, that absence of positive evidence that a treatment works becomes fairly powerful evidence that it doesn’t work. In any case, this “more fluid concept of evidence” clearly involves patient anecdotes, which are, for the many reasons explained here over the years, often unreliable and even misleading. That is why they are relegated to a much lower rung on the ladder of evidence than clinical trials.
As you might imagine, several of the longtime bloggers at SBM have discussed Dr. Katz’s idea of “a more fluid concept of evidence,” including Steve Novella, Kimball Atwood, and myself. More recently, Dr. Katz was very unhappy over how Steve Novella and I had criticized the Cleveland Clinic for opening a traditional Chinese medicine clinic run by a naturopath. I have characterized David Katz’s attitude as a false dilemma: Abandon patients or abandon science. And so it is. Unfortunately, it is a false dilemma that lies at the very heart of the very concept of “integrative medicine,” as you will see.
Dr. Katz lashes out again
Dr. Katz is once again very unhappy with us over here at SBM. One can’t help but notice right away how he refers to us (without naming us, although he does at least link to us) as the “self-proclaimed guardians of the true definition of evidence-based medicine” (EBM). Ouch. He sure makes us sound as though we are uptight, doesn’t he? Except that we don’t really do that. Rather, we advocate changing the definition of EBM to be more like SBM because of the obvious defects in the definition of EBM that allow pseudoscience to infiltrate medicine. It’s not as though we haven’t discussed this more times than I can remember.
I get it, though. No one likes being criticized or having his favorite ox gored. I can even sort of understand why Dr. Katz seems to think that we are a bunch of peons unworthy to criticize him, as once again he has to lay down his bona fides in a rather obvious appeal to authority:
I opened an Integrative Medicine Center some 15 years ago, and have directed it, and seen patients in that context, since. I did so not because of any long suppressed desire to practice medicine under that banner, and certainly not because of any inclination to don Birkenstocks.
In fact, I was — as I remain- a card carrying member of the evidence-based medicine club. I was already running then, as I still am, a federally funded clinical research lab. I was already then, as I am now, routinely publishing studies in the peer-reviewed literature. I was teaching then, as I did for roughly ten years, biostatistics and clinical epidemiology to Yale medical students. And while back then I had co-authored a textbook on epidemiology and biostatistics, I have since co-authored four editions of that textbook, and a textbook on evidence-based medicine as well.
In other words: How dare you nothings criticize me? Yes, Dr. Katz has been using variants of this same whine since he dismissed David Colquhoun, Steve Novella, Dr. R. W. Donnell, and myself as not having “earned the right to express an opinion” on these issues and of “simply having an opinion, an Internet connection, and some time to kill.” He even referred to us as “health hazards of the blogosphere” and quacks.
Although not to the extent that Dr. Katz has, I, too, have been federally funded and I, too, routinely publish in the peer-reviewed literature. Unfortunately, I probably stuck too closely to basic science too long before transitioning it into the realm of clinical research, given that a surgeon will always have a great deal of difficulty directly competing with basic scientists at basic science, hence my difficulties retaining external funding more recently. In retrospect, I’m amazed I did as well as I did for as long as I did competing with basic scientists and am hopeful that my recent forays into much more directly clinically relevant areas are not too late, given my priority score on a recent grant that just missed the draconian cutoff for funding in today’s—shall we say?—highly constrained funding environment.
Of course, Dr. Katz’s impressive qualifications, which I concede are more impressive than mine, do not matter much when he routinely spouts nonsense such as his justification of the use of homeopathy, naturopathy, and traditional Chinese medicine, as he has unfortunately done so often in the past. Only the quality of his arguments for the “integration” of various alternative medicine practices into conventional medicine matter, and these arguments have been uniformly of very poor quality, as we have documented for over seven years now. This time around is no different.
I’ll start with his third complaint, because it is a complaint so frequently used by advocates of quackery (whom, whether Dr. Katz realizes it or not, he is emulating by repeating that complaint), which is all about the money:
And third, there is the thorny little issue of money.
The horses that pull toward “evidence” pull a cart full of money. The average cost of bringing a new FDA-approved drug to market now approximates a billion dollars. Spending millions, tens of millions, or even hundreds of millions on clinical trials makes sense if a return in the billions is the offing. It’s pretty much a non-starter otherwise, and a problem even a budget the size of the NIH’s cannot solve. In fact, the price tag for one, new FDA-approved drug is nearly ten times the annual operating budget of the National Center for Complementary and Integrative Health.
Hmmm. Where have we heard this one before? I wonder…
Oh, yes. It becomes clear when Dr. Katz proceeds to use his favorite tale to illustrate how money supposedly rules everything in medicine. Basically, it’s a tale of co-enzyme Q10 (CoQ10), a tale he’s told many times before going back to his original “more fluid concept of evidence” talk in 2008, with periodic repetitions, up to and including Katz’s current post. CoQ10 is also known as ubiquinol and is found widely in plants. It acts at the level of the electron transport chain to facilitate the production of chemical energy through oxidative metabolism in our mitochondria. According to Dr. Katz, we tend to get most of what we need from our diet because our bodies don’t make CoQ10 very efficiently, although a recent review calls into question Dr. Katz’s claim that we get most of our CoAQ10 from our diet rather than our body’s own synthesis of it.
In any case, it was thought that CoQ10 would be a good adjunct in heart failure, in order to improve the function of the heart. In 2000, there was a small randomized double-blind placebo-controlled clinical trial that concluded that CoQ10 “does not affect ejection fraction, peak oxygen consumption, or exercise duration in patients with congestive heart failure receiving standard medical therapy,” for which investigators concluded “we detected no objective benefit from coenzyme Q10 administration in patients with heart failure” and that their “study shows no benefit to adding coenzyme Q10 to the standard treatment of heart failure.” Dr. Katz was very unhappy about this and characterized this study as being used to “drive the final nail in the CoQ10 hypothesis,” while mentioning that within a year a large RCT of a prescription drug carvedilol (trade name: Coreg) was published showing that it was safe and effective for the treatment of heart failure. To Dr. Katz, this is the evidence of big pharma primacy and the premature rejection of a promising treatment.
He then goes on:
It took more than a decade after that to undo the damage, but eventually it happened: trial data came in to show that co-Q10 could reduce heart failure mortality by about 50 percent. In other words, ten years after being declared defunct, the co-Q10 “hypothesis” was not only vindicated, but seemed to represent the greatest advance in the treatment of heart failure in years.
First of all, I can’t help but note that if the CoQ10 hypothesis was truly dead, then why was a relatively large randomized, double-blind study begun a mere three years later? Something doesn’t jibe here. If the 2000 study had truly “closed the door,” then no IRB would have approved a bigger study. Clearly, the forces of big pharma were not as powerful as Dr. Katz insinuates. I also find it odd that he didn’t actually link to the primary study by Mortensen et al, which was published in JACC Heart Failure in December. In any case, one thing that I noticed about this study right away is that there were no differences in short term endpoints; so in that, at least, this study by Mortensen agrees with the 2000 study. It was only in long-term outcomes that it showed a significant difference, with a major decrease in mortality.
The other thing that I noticed is that, judging from the introduction and discussion by Mortensen et al, Dr. Katz appears to be giving a highly biased account of what happened. There were clearly a lot more uncertainty and conflicting data than his accounts would leave you to believe, and, of course, it was completely appropriate for the original study authors to conclude that, based on their results, it was not appropriate to recommend CoQ10. That’s a provisional conclusion justified by the study at the time. That’s the way clinical science progresses after preclinical validation. Pilot studies are done. If they are promising, then small RCTs are done. If those are promising (and sometimes if they are mixed, as judging from the introduction and discussion of this paper), then larger RCTs are done. There’s nothing outrageous here, and Mortensen et al also acknowledge that their study might have enrolled a larger proportion of patients with milder symptoms than other trials.
In an accompanying editorial, Justin Ezekowitz notes that there were accrual problems, which is why the study took ten years, noting that it’s “unclear whether this situation resulted from patients’ acceptance of the experimental therapy, the ability of the sites and investigators to conduct a trial, the intensity of follow-up, or other features that are not further elucidated.” He also notes that it is premature to recommend CoQ10 as part of standard guidelines for treating heart failure, noting that these “striking results should be interpreted with caution given the small population and event numbers and the large treatment effect; many of these results were not replicated when a subsequent adequately powered trial was done (11).”
Note that the reference refers to a study by John Ioannidis that found that smaller studies generally produce large treatment effects that decline when larger trials are done.
To complete my deconstruction of Dr. Katz’s complaint about no money for studying supplements like CoQ10, I like to echo Dr. Ezekowitz’s remonstration with Mortensen et al:
Enthusiastically, the investigators have said that CoQ10 “should be added as standard therapy” based on these results, and additionally, it is a “natural and safe substance” (12). Perhaps this second statement is driven by the fewer side effects in the CoQ10 group than in the placebo group (13% vs. 19%). Further study in adequately powered clinical trials is required because it is premature to suggest that this finding reaches the necessary efficacy or safety bar required to consider prescribing this drug. Additionally, although CoQ10 is widely available and taken as a supplement, safety cannot be assumed for any drug in this context, and so much broader and more detailed experience is required to declare this substance acceptably safe. Finally, CoQ10 is natural, naturally made in a factory by pharmaceutical-grade processes of medicinal chemistry and yeast or bacterial fermentation.
Indeed. What was that about Dr. Katz denying that he believes that “natural is better,” again? After all, if CoQ10 does “work,” improving outcomes in heart failure—and it very possibly does, given the results of Mortensen’s trial—it’s working as a drug, and the actual supplement used is manufactured in a manner very much like the methods used to manufacture pharmaceuticals. In particular, I like how Dr. Eskowitz fires a shot across the bow of advocates like Dr. Katz and the supplement manufacturers:
Nutraceutical manufacturers must step up and, like pharmaceutical and device companies, fund a trial for which an indication is sought or advertised, not solely through governmental funding agencies. If 1% of the estimated $1 billion annual market for CoQ10 was reinvested into an adequately powered, appropriately designed, academically led trial to demonstrate the efficacy and safety of CoQ10, then perhaps we would have a scientific advance worth being excited about.
Yes, the lack of patentability is a problem, but it’s not an insurmountable one. The problem is that supplement manufacturers want it both ways, and, thanks to the DSHEA of 1994, a law that basically lets supplement manufacturers get away with all sorts of vague health claims without evidence, they can have it both ways. Why would they want to spend even 1% of their profits and subject themselves to FDA pharmaceutical regulation if they don’t have to? There’s more greed over nutraceuticals than just big pharma and the lack of patentability. Funny how Dr. Katz never seems to acknowledge that.
Integrative medicine “individualization”
Now let’s go back to Dr. Katz’s “more fluid concept of evidence,” which, given his arguments, sure sounds like an excuse to change the rules of evidence in the middle of the game, as Steve has charged. In the case above of CoQ10, he argues that if only there had been a more “fluid” concept of evidence applied to the original 2000 trial, then maybe 13 years might not have been “wasted” and we might have been able to incorporate this miraculous supplement into treatment regimens of people with heart disease. I view it more as a cherry picked example, of Katz being lucky in probably having guessed right. How much more often do clinical investigators guess wrong after an initially-promising clinical trial? That’s what the whole concept of the “decline effect” is about, where initially-impressive effect sizes tend to decline with time as further studies are done. As I like to say, the “decline effect is science correcting itself over time.
Be that as it may, Dr. Katz, not satisfied with complaining about funding and nefarious pharmaceutical companies, launches into what I like to call the “personalization” gambit:
First, the evidence from any given clinical trial, however methodologically robust, may or may not pertain to any given patient. The application of evidence from trials to the care of individuals not in those trials is itself a product of the art of medicine, not the science. To my knowledge, there has never been a randomized trial examining different ways clinicians might decide if trial results pertain to a specific patient. That may result from the developing field of pharmacogenomics, but we have a ways to go. In other words, for now, the application of the evidence in evidence-based medicine to the actual care of patients is not, itself, evidence-based. That’s a fact.
No, actually, it is not a “fact.” It is Dr. Katz’s opinion. Also notice how Dr. Katz subtly attacks a straw man version of EBM by emphasizing only RCTs, implying that nothing is known because there hasn’t been an RCT of applying evidence-based practice (EBP) guidelines. For one thing, it’s just not true that nothing is known about the application of EBM. True, not enough is known, but a lot is known. Moreover, there have been studies of EBP guidelines, both prospective and also both simulated and in the real world. Has Dr. Katz ever heard of a surgical checklist? These are checklists that must be done before surgery to ensure that various EBP items on the list are “checked off,” thus ensuring adherence to EBP guidelines. No doubt Dr. Katz would claim that these are not applicable because they are used in a much more controlled, regimented world and don’t show how physicians apply guidelines to individual patients, but surgical patients are individual patients and checklists allow for not performing an intervention if it is deemed inappropriate for a given patient.
I also can’t help but note that in the case of CoQ10, Dr. Katz seems rather eager, based on one RCT, to adopt a “one size fits all” guideline recommending the addition of CoQ10 supplementation to the treatment of heart failure patients.
Be that as it may, Dr. Katz is also attacking a straw man. No one—and I mean no one—arguing for SBM and against the sort of integrative relativism espoused by practitioners like Dr. Katz claims that randomized clinical trial (RCT) results must always apply to a given patient with a disease presenting for evaluation. That’s why it’s called science-based medicine. Heck, that’s why EBM was originally referred to as evidence-based medicine. That’s because, ideally, medicine is based in science and evidence, but science and evidence are not the be-all and end-all.
What the patient values, for example, matters. For example, I routinely tell women diagnosed with breast cancer that mastectomy and lumpectomy followed by radiation therapy both produce equivalent survival rates. In women for whom it is appropriate, I generally recommend breast-conserving surgery (lumpectomy) first. However, even though the survival rates are the same for the two procedures, there is a somewhat higher chance of a local recurrence (recurrence in the breast) with breast-conserving surgery. The reason that that local recurrence doesn’t portend a higher risk of death is because such local recurrences can be “salvaged” with mastectomy. Some women have a very hard time dealing with even the relatively low risk of local recurrence inherent in breast conserving surgery even if the survival is the same. There’s also the issue that radiation therapy, as given, can be rather inconvenient. Most regimens involve daily treatments for four to six weeks (it used to be always six weeks), which can be a pain if the woman doesn’t live close to a center with a radiation oncology facility.
So is it “wrong” to do a mastectomy to treat a small breast cancer that could be treated with breast-conserving surgery? Of course not. I admit that I will at least try to persuade a woman whom I deem a good candidate for breast conserving surgery out of a mastectomy because it’s a much bigger operation, but if she insists on mastectomy I will do it. The point here is that informed consent is a critical component of EBM and SBM, but that informed consent must be truly informed. In other words, the information given to the patient must be accurate and based on the best science available. That’s where integrative medicine fails, both as EBM/SBM and ethically. I’ve frequently referred to the concept of “misinformed consent.” What integrative medicine does is to introduce treatments not based in science as though they were.
Dr. Katz then invokes a common scenario in medicine:
Second, the needs of patients all too often go on when evidence runs thin, or out entirely. This is why I went into “integrative” medicine in the first place — and why I am now involved in the online delivery of holistic care to a larger population.
By working with colleagues trained differently than I, naturopathic physicians in particular, we had more treatment options together than we did alone. When patients had been everywhere, tried everything subtended by RCTs and still weren’t better, they came to us. And we, generally, could come up with something reasonable to try. Admittedly, it wasn’t yet in the textbooks — but that was the point. These were the folks that had already run off the pages of textbooks, and still needed help.
This, of course, is another straw man view of EBM and SBM. Indeed, one of our criticisms of EBM is similar, but different, than Dr. Katz’s. I like to refer to it as methodolatry, defined as the profane worship of the randomized clinical trial as the only valid method of investigation. So there we agree! The RCT should not be the be-all and end-all of investigation! After all, there will never be large, double-blind RCTs to answer every medical question that needs answering. Sometimes it’s unethical to do an RCT. Sometimes it’s too impractical, either because of expense, time, or both. Sometimes the condition is just so uncommon that there will never be enough patients to do an RCT. For instance, at the recent Society of Surgical Oncology meeting in Houston, it was discussed how doing an RCT comparing outcomes in patients who undergo standard pancreatectomy to those who undergo minimally invasive pancreatectomy (laparoscopic or robot-assisted) for pancreatic cancer with overall survival as the primary outcome would require over 6,000 patients and several years. In such cases, we are forced to use the best evidence available, because sometimes the necessary trials will never be done.
Here’s the problem, and here’s where Dr. Katz and I diverge. We at SBM advocate that patients are best served when they are not subjected to unscientific and pseudoscientific medicine and when the treatments recommended to them are firmly rooted in science. Just because there is a lack of evidence regarding the best management for a given patient does not mean you can scientifically or ethically justify recommending pseudoscientific treatments like homeopathy as though they have evidence to support them! Irish comedian Dara O’Briain has a brilliant retort to this kind of argument, which is basically of the same general form of (medical) “science doesn’t know everything”:
Jesus, homeopaths get on my nerves with “science doesn’t know everything.” Well science knows it doesn’t know everything. Otherwise, it’d stop…But just because science doesn’t know everything doesn’t mean you can fill in the gaps with whatever fairy tale most appeals to you.
That’s exactly what integrative medicine does by advocating the use of modalities such as reiki, homeopathy, naturopathy, and much of traditional Chinese medicine. And that’s exactly what Dr. Katz is doing when he advocates a “more fluid concept of evidence.” Don’t believe me? Behold these words from Dr. Katz:
Nowhere in the argument for a more fluid application of evidence to the care of people is there an argument to reject evidence when it is clear. The evidence in support of immunization is clear. A connection between the two argument is specious; a non sequitur.
Note that apparently Dr. Katz was criticized that his a more “fluid” concept of evidence has been used by the antivaccine movement. Now contrast the above passage with these words from Dr. Katz:
And the anecdotal evidence that homeopathy might be effective was brought up by the naturopaths. We tried it. It worked.
If there’s a form of quackery other than homeopathy for which the evidence is more clear that it does not work better than a placebo, that, physically according to what we know from chemistry and physics, it cannot work, I am unaware of it, with the possible exception of reiki. In 2008, Dr. Katz was clearly willing to reject evidence that homeopathy is pseudoscience. What other clear science has he been willing to reject when he sees fit? How “fluid” is his concept of evidence? In the final section, we shall see.
Dr. Katz’s lovefest with naturopaths
Dr. Katz works a lot with naturopaths. We’ve known this for nearly eight years. Apparently, his definition of evidence is “fluid” enough to embrace a specialty with as tenuous a relationship with science as naturopathy whole-heartedly, to the point where he’s recently given testimony to the Public Health Committee in the Connecticut State House of Representatives in support of a bill that could expand the scope of practice of naturopaths and—or so naturopaths hope—give them prescribing privileges. In his written testimony, he brags about how he has “worked extensively with naturopathic physicians in every context relevant to the advancement of medicine” and gushes shamelessly about how he has been “richly rewarded by all of these collaborative activities.” He even refers to naturopaths as “ideally suited to meet the primary care needs of Connecticut residents.”
What is it about naturopaths Katz so admires that he’s willing to go to bat for them in front of the legislature in his own state and speak of them in such glowing terms? Is it their use of quack treatments, such as unsupported thyroid tests and treatments for fake diseases like “systemic candidiasis” and Chronic Lyme disease? Given Katz’s admirable defense of vaccines in wake of the Disneyland measles outbreak, is it their near-universal antivaccine views, even among naturopaths who think of themselves as “not antivaccine”? Is it their advocacy of major quackery, including chelation therapy, a potentially deadly pseudoscientific treatment, to treat autism? Is it their advocacy of intravenous peroxide to treat a wide variety of conditions? Is it their advocating “detoxification,” herbs, and acupuncture to treat infertility? Is it how naturopaths embrace the four humors? Is this what he means by embracing a “more fluid concept of evidence”? Apparently so. After all, the humors were believed to be fluids.
Or perhaps Dr. Katz really doesn’t know what naturopathy is actually about. If that’s the case, I’ll take the opportunity to educate him and suggest that he read Kimball Atwood’s classic critical appraisal of the vitalistic pseudoscientific mystical practice that is naturopathy, after which he might be interested in what naturopaths say amongst themselves when they think no one is listening. Then he should read Britt Hermes’ confessions of a former naturopath and her description of naturopathic education. Let’s just say, the facts about what naturopathy is and how naturopaths practice contrast starkly with the naïve view Katz presents in his testimony. A lot.
Seven years after he introduced his more “fluid” concept of evidence, Dr. Katz has failed to give a satisfactory or convincing answer to the question of just what he meant. The best he can come up with, after whining about how nasty we “self-appointed guardians of the definition of EBM” with an Internet connection and too much time on our hands are, seems to be either to substitute anecdotal evidence for RCT evidence whenever he thinks it appropriate or to seek a “middle way” between SBM and quackery. If that means embracing homeopathy, traditional Chinese medicine, and naturopathy, apparently that’s fine with him. Unfortunately, Dr. Katz is not alone. He is on the vanguard of a whole movement advocating a more “fluid” application of evidence who, with good intentions, have brought rank pseudoscience into bastions of academic medicine.
Additional reading:
     1.  Changing the Rules of Evidence
     2.  The false dilemma of David Katz: Abandon patients or abandon science
     3.  Science, Reason, Ethics, and Modern Medicine, Part 2: the Tortured Logic of David Katz
     4.  “Integrative” medicine at Yale: A more “fluid” concept of evidence?

Boundaries
Mark Crislip on April 5, 2013
Link
Vacation then taxes have consumed my focus the last two weeks, and I have had little time to devote to issues of infectious diseases, much less SBM, so I will instead meander around a more philosophical terrain. I feel guilty when I do not have a substantive, data driven post evaluating a paper or essay in detail, but some weeks there just is not the time.
Being involved with SBM has been, like all intellectual endeavors, a process rather than result. I keep experimenting with conceptual frameworks with which I can understand the differences between a SBM approach and a SCAM (supplements, complementary and alternative medicine) approach. Nothing clarifies thoughts quite like writing them down. Or maybe not.
The motto of the blog is “exploring the relationships between science and medicine” but it is often more about non-overlapping boundaries* than relationships. We are often separated more by Berlin walls than Venn diagrams.
There are perhaps four boundaries that separate science-based medicine from those who prefer SCAMs. More if you are a splitter; I am a lumper by nature. At work I am an Occam kind of guy.
One boundary separates reality from fantasy. Reality is defined by the description of the natural world provided by the sciences. This blog keeps pounding away at the wall, with remarkably little effect. Be it anti-evolution, anti-vaccine or any other non-reality-based world view, they appear immune to the facts.
It is weird, and it took me a long time to realize that for many people facts are just not important. For 30 years I have worked in medical teaching institutions where facts are important. I see a patient where the intern has treated a methicillin susceptible S. aureus bacteremia with a short course of IV vancomycin and then changed the patient to oral clindamycin. I see a similar case or two at the start of almost every academic year. I go though all the permeations of S. aureus infections and the antibiotics used to treat them, going through all the clinical trials to justify my conclusions. Just the facts ma’am.
And when we are done the intern will not do THAT again. Trust me. No one wants to sit through me talking about S. aureus more than once in a career. But in medicine the facts change what we do. Eventually. That never seems to be the case in the SCAM world. It took me years to understand that although my blog entries often focus on the facts of a topic, reality as I understand it, for many, if not most in the SCAM community, facts do not matter. Facts only serve to support a predetermined world view, not alter it.
This month an article was published by the CDC looking at whether autism is caused by too many vaccines in too short a time, the “too much too soon” hypothesis. Dr. Gorski covered it a few days ago.
From basic principles one would not expect the paltry number of antigens in vaccines to be associated with autism, and the study suggests there is NO relationship between the number of vaccines and the development of autism.
I was looking forward to an essay at Age of Autism with the title: “We were wrong. Too many too soon is a failed hypothesis.” I should also look forward to an end to global warming and increased Medicare reimbursements.
To my knowledge, those who are the topics of this blog never change their beliefs as a result of high quality medical studies, which is hard to do even if you are not emotionally invested in the concepts studied. The longer I practice medicine the more effort it takes to practice differently based on new information. In the SCAM world they are immune to reality-based change, which, in the case of anti-vaccine proponents, is a curious phraseology.
Another boundary is philosophical/political/moral. It is a weird wall that separates us. I can understand the wish to be free to make your own decisions about your health care. What I can’t wrap my head around is the conspiratorial nature of many of the comments on this and other blogs. That there is a cabal of Government, pharmaceutical companies and doctors conspiring to hide the truth from people. And we do it because we are not only greedy, profiting from the medical-industrial complex, but we have evil motives. We are bad guys.
It is a world view that I overheard on vacation. I was wandering around a jewelry shop in Santa Fe and I overheard a customer telling the poor cashier that “they know what causes cancer and how to cure it, but there is too much money in the status quo.”
I do not know how common that world view is, which to my mind is as fantastical as the precepts of homeopathy, but it is a loud one. The conspiracy is always the last refuge of those who cannot accept the findings of science, as Orac points out:
They would never publish a study showing vaccines cause autism.
Sure they would. That’s what the CDC does. That is what they have done and always will do.
I do not comprehend the conspiratorial approach to the world. I tend to see most people in health care, real and alternative, as doing what they think is the right thing for people. I do not see SCAM proponents as evil, just misguided. They certainly do not see SBM in the same way.
Another boundary is in understanding that the fundamental flaws in thinking that define human behavior, the logical fallacies, apply to everyone all the time: me, you, all of us. Most of the time proponents of SCAMs fail to comprehend my favorite observation by Richard Feynman:
“The first principle is that you must not fool yourself and you are the easiest person to fool.”
It is odd that those us of at SBM may have the adjective “arrogant” applied to us, but I think we are just the opposite. We are all too aware of the numerous ways we can think poorly and be fooled. That ability seems lacking in proponents of SCAM.
So often when reading a study that allegedly demonstrates efficacy of one SCAM or another there are flaws that allow the researcher or the patient to fool themselves. It is remarkable how often these flaws go unnoticed.
The final boundary is not realizing that the usual standards by which we evaluate the medical literature do not apply when evaluating the alleged effects of SCAMs. The rules of engagement are slightly different for reality- than for fantasy-based interventions.
Studies of the effects homeopathy or reiki or acupuncture are not evaluations of a therapeutic intervention that has an effect on an anatomic structure or a physiologic process. They are best seen as elaborate psychological evaluations of the reaction of people to complicated artificial rituals. They are medical equivalent of the Stanford prison experiment where patient and doctor follow explicit and subtle cues to behave as if something is occurring that could alter a disease.
When evaluating nonsense in a clinical trial, are you doing science? Yes, just not good science. If the principals you are studying are based on fantasy, like homeopathy or chelation, it is mummery of science; it is tooth fairy science.
I was struck this week by a quote of one of the TACT investigators and their inability to recognize in TACT tooth fairy science combined with methodological flaws, as the flaws were noted by skeptics:
“Although skepticism has an important role to play in critical debates, it is easy to overplay that hand. The people you mention seem to have a very naïve view of science, very far removed from the messy realities of daily work of people doing science. It is also important to remember that even the most hard core scientists can have some pretty eccentric views when they venture outside their narrow field of expertise. What does that imply about science and the people who wish to guard its borders?
There has been a project in philosophy to identify firm demarcation criteria that will allow for a distinction between science and pseudoscience. While some useful work has resulted, the overall attempt failed. Gets into some deep waters, but the harder the philosophers tried to find that electrified fence that marked off “real science” from the rest of human thought, the more they undermined the borders of science itself. Interestingly, “real scientists” rarely worry about whether they are doing science. They consider the question uninteresting, leaving it for the philosophers, sociologists and (now) the bloggers!”
For three years as a fellow I tried being a “real scientist”, involved with investigating the interactions of Candida and endothelial cells and what made the former adhere to the latter. I was involved in a clinical trial using anti-endotoxin antibody to modify the course of sepsis. I was the Peter Principal personified. It is with good reason I am a clinician.
In those days I never worried whether I was doing science, because Candida and endothelial cells are tangible, as are endotoxin and antibodies.
I did not try to generate adherence negative Candida mutants with N-Ray’s; I used UV. I did not suppress bacteria in the endothelial cultures with homeopathy, I used antibiotics. I did not try to alter the pathophysiology of sepsis by altering qi, I tried to inactivate bacterial endotoxin. I kept within the bounds of reality.
The question is not whether “real scientists” are doing “real science” but are they doing good science: careful evaluation of complex, and real, phenomena where potential biases and flaws are known and controlled for.
When you study a real intervention against a real phenomena, you need not worry quite as much about the potentially confounding issues raised by fantasy. When you are studying nonsense as if it were real, those potentially confounding biases become of paramount importance.
What skeptics recognize are “the messy realities of daily work of people doing science” especially as they apply to the scientific evaluation of fantasy-based therapies. Just as the best investigator for psychic powers is a magician, not a physicist, the best critic of a study of nonsense is a skeptic. We are experienced in recognizing the errors that most people either do not consider or fail to apply to themselves.
It is an ongoing curiosity how real scientists become blind to the confounding flaws and even reality when they study SCAMs. So many, where read carefully, are all sound and fury signifying nothing.
There many boundaries between SBM and SCAM. They all seem thicker and taller than when I started blogging years ago. Then I thought all that people needed to know was the difference between reality and fiction. A wee bit of education. Let them know why this SCAM or that is nonsense and they would say “oh, I see” and we could all move on. Naïve and simplistic, I know. But that is the story of all my intellectual endeavors. The more I know, the more complicated it becomes.
Sisyphus had it easy.
*no way am I going to say Non-overlapping magisteria ; je déteste ça.

Conspiracy Thinking And Alternative Medicine
Steven Novella on October 2, 2024
Link
Just recently, in casual conversation, someone commented to me that “they” already have a cure for cancer but are hiding it. This is a claim I frequently encounter. One survey, from 2005, found that 27.3% of those surveyed endorsed this belief. Another survey from 2014 found that 37% of people believe the FDA is suppressing natural cancer cures on behalf of “Big Pharma”. A 2018 survey found that 37% of people believe that cancer can be cured solely through natural alternative treatments.
These two claims – that there is a conspiracy to suppress already known cancer cures and that alternative treatments can cure cancer, are related. They are representative of the symbiotic relationship between the conspiracy subculture and the CAM industry.
This relationship is perhaps most disturbingly and cynically represented in the persona of Alex Jones. According to many sources, the Info Wars empire exists solely to sell snake oil. That’s really it – it is a dietary supplement business that uses conspiracy mongering as a marketing tool. The conspiracy messaging has been market tested in every broadcast, and the message tweaked to maximize sales. Jones apparently hit upon a pre-existing relationship, and then turbo-charged it with the heartless precision that only a con-artist can (without the baggage of actually believing anything).
That there is a huge overlap between CAM true believers and conspiracy theorists is not surprising. Both are the natural consequence of rejecting science, the institutions of science, and expertise. The thought process flows naturally. If, for example, we begin with a claim – that coffee enemas cure cancer – proponents of this claim run into a hard brick wall of fact. Coffee enemas do not cure cancer. Within a scientific framework, for various claims, we might say that there is no credible evidence the treatment works, or that there is no plausible mechanism, or that direct clinical evidence does not show any benefit, or that there are risks far in excess of any potential benefit. In short, we use science to answer the question – is this intervention safe and effective.
This is a problem, however, if you want to sell coffee enemas to cure cancer, or use the claim to sell your services as a health guru. So what’s a snake oil purveyor to do? The pseudoscience handbook provides an answer – just dismiss the scientific evidence as part of some conspiracy. “They” don’t want you to cure your cancer with coffee enemas so that they can keep you sick and sell you their drugs.
Unfortunately, the conspiracy narrative is inherently emotionally compelling to the human brain. Some people seem to be innate conspiracy theorists, and buy pretty much every conspiracy. That is just how they see the world. But there is a continuum, and at the other end there are those with more of an analytical thinking style who are relatively resistant. But most people are in between, and are “opportunistic” conspiracy theorists – they will latch onto conspiracy theories that support their existing belief system or their tribal identity.
Conspiracy thinking is also a trap because it contains feedback loops of positive reinforcement. If you reject authoritative sources of information as part of a conspiracy, then you will likely reject any information that can disprove the conspiracy. Any information that contradicts the conspiracy, or lack of information that would prove the conspiracy, are part of the conspiracy. A shadowy “they” then hovers over all information, which gives you permission to reject, ad hoc, any inconvenient data point.
Conspiracy thinking can also be emotionally appealing. The conspiracy believer sees themselves as part of a privileged enlightened few, who can see the man behind the curtain. The rest of the world is divided to dupes and villains, but they are the army of light, the Rebel Alliance who knows that the Empire is secretly being run by a Sith Lord.
It also gives them permission to sell snake oil without the burden of due diligence, quality control, doing actual research, making difficult decisions, or compromising profit in the name of good patient outcomes. This is why pretty much every CAM claim comes wrapped in a conspiracy theory, and every conspiracy website sells snake oil supplements on the margin. The supplement industry is financing the conspiracy industry, while the latter is marketing for the former.
Social media has, unsurprisingly, also supercharged this dark symbiosis. It has elevated another element to the conspiracy thinking dynamic – radicalization. While yes, there are all-purpose conspiracy theorists and opportunistic dabblers, social media algorithms have created feedback loops that radicalize the latter into the former. Conspiracy theorists, it turns out, are reliable content consumers. If you want to drive clicks, radicalize your viewers.
The media has always been a willing partner to the conspiracy theory – alternative science dynamic. Such narratives serve their purpose of sensationalism. Social media did not invent click bait, it just brought it into the 21st century and automated it with algorithms.
The end result is that media driven, snake oil financed, conspiracy thinking is rotting out the core of our society. It is also compromising the quality of our health care, luring people away from complicated but at least tested and standardized interventions to harmful and expensive fantasies. It is creating a generation that does not even recognize expertise, that sees all claims as equally valid, in fact validity does not even come into it. Claims are all performance, and exist solely to generate social media engagement.
We are now also a hairs-breath away from a dangerous medical conspiracy theorist possibly being put in charge of our institutions of health – RFK Jr. He had his own journey of conspiracy radicalization, and now wants to effectively run the NIH, CDC, and FDA, and install like-minded conspiracy theorists to head these organizations. It’s difficult to imagine a more horrific disaster for American healthcare than this scenario.
But this is where the unholy alliance of conspiracy theories and snake oil has brought us.

Medical Conspiracies
Steven Novella on March 19, 2014
Link
Anyone publicly writing about issues of science and medicine from a pro-science perspective likely gets many e-mails similar to the ones I see every week. Here’s just one recent example:
Im sorry the medical community has become decadent and lazy as most that follow your stance could care less to study the real truth. I have also seen it much more deviant as many professionals know the risks and harm vaccination cause but continue to push it through there practices because of pure greed. Many are also scared of loosing there practices for not following the corrupt industry. Im sorry but the medical industry has become drug pushing decadent slobs that only care about there bottom line.
The e-mailer clearly has a particular narrative that he is following (in addition to the amusingly common poor grammar and spelling). He even writes at one point in our exchange, “the details really don’t matter at this point what matters is what the bigger picture…” He is certain of his big picture conspiracy narrative. The details are unimportant.
Being on the receiving end of an almost constant barrage of such medical conspiracy theories it might seem that such beliefs are extremely common. Of course, such e-mails are self-selective and therefore not representative of the general population. I was therefore interested to see a published survey polling the general population about such beliefs. The survey is published in JAMA Internal Medicine, authored by Eric Oliver and Thomas Wood.
Here are the six survey questions and the percentage who agree or disagree (the rest indicating that they do not know).
The Food and Drug Administration is deliberately preventing the public from getting natural cures for cancer and other diseases because of pressure from drug companies. (37% agree, 32% disagree)
Health officials know that cell phones cause cancer but are doing nothing to stop it because large corporations won’t let them. (20% agree, 40% disagree)
The CIA deliberately infected large numbers of African Americans with HIV under the guise of a hepatitis inoculation program. (12% agree, 51% disagree)
The global dissemination of genetically modified foods by Monsanto Inc is part of a secret program, called Agenda 21, launched by the Rockefeller and Ford foundations to shrink the world’s population. (12% agree, 42% disagree)
Doctors and the government still want to vaccinate children even though they know these vaccines cause autism and other psychological disorders. (20% agree, 44% disagree)
Public water fluoridation is really just a secret way for chemical companies to dump the dangerous byproducts of phosphate mines into the environment. (12% agree, 46% disagree)
The numbers are not surprising, in fact I would have guessed they were a bit higher, but again that perception is likely distorted by my e-mail inbox. They found that 49% of Americans agreed with at least one conspiracy, and 18% agreed with three or more. This is in line with the level of belief in non-medical conspiracies. They did not publish, but I would be interested, in the percentage of people who said they disagreed with all of the conspiracies. Many of the respondents indicated that they did not know if a particular conspiracy were true, likely because they had not heard of it before, but were unwilling to disagree on plausibility grounds alone.
An earlier study by Oliver and Wood found similar percentages for political conspiracies. In that study they concluded that belief in conspiracy theories does not track with any particular ideological belief, but rather with a, “willingness to believe in other unseen, intentional forces and an attraction to Manichean narratives.”
This would seem to be true for the 18% who believe in three or more medical conspiracies – they have a tendency to believe in conspiracies. For those who believe in only one or two conspiracies, that might have less to do with personality and more to do with culture and ideological beliefs.
The second part of their study is perhaps more interesting. They found a strong predictive correlation between belief in the above conspiracies and a host of medical behaviors. Conspiracy believers were more likely to use herbal supplements, use alternative medicine, and eat organic food, and less likely to vaccinate, use sunscreen, and have regular physicals.
The largest effect sizes were for taking herbal supplements, which went from 13% for zero conspiracy beliefs to 35% for three or more. Buying from local farms was similar, going from 14% to 37% respectively, while using sunscreen only decreased from 38% to 30% respectively.
I would have been interested in seeing other responses, such as the percentage who visit a naturopath or chiropractor. I would predict that more hard-core use of clearly “alternative” practices (more than just taking supplements) would be a strong predictor of belief in medical conspiracies. This is based upon the observation (documented in many of the articles here on SBM) that much CAM promotion is intimately tied with medical conspiracy thinking. Anti-vaccine sites, for example, often promote homeopathy and other alternatives to vaccinations. Mike Adams of naturalnews is perhaps the most dramatic example of the convergence of CAM and all manner of conspiracy theories.
Conclusion
This survey by Oliver and Wood indicates that belief in medical conspiracies is fairly common, and that such beliefs are not benign, but correlate negatively with important medical behaviors. It’s difficult to tease out cause and effect, and this survey makes no attempt to do so. It’s possible that the same personality profile is attracted to both conspiracy theories and alternative medical practices. It’s also possible that the CAM subculture encourages belief in conspiracy theories, and that the conspiracy subculture encourages rejection of mainstream medicine and acceptance of fringe ideas. I think all of these factors conspire together to create the effect we are seeing in this survey.
The medical community would be well-served if they understood the phenomenon of medical conspiracies. In fact, it can be viewed and addressed as a public health issue. Medical institutions can take such beliefs more seriously, rather than just dismissing them as fringe. Efforts to educate the public about critical thinking, scientific methodology, and how the institutions of medicine work and are regulated, might reduce the popularity of such conspiracy theories.
I also think we need to have as much transparency as possible in scientific and regulatory processes. Secrecy or even opaqueness tends to breed paranoia.
This data (if I may indulge in a little self-promotion) also highlights the importance of efforts such as science-based medicine, the goal of which is to popularize understanding of the science of medicine, and taking a critical view of popular misconceptions, including medical conspiracy theories. I would argue academic and medical institutions to take such efforts more seriously.

Conspiracy Medicine
Steven Novella on July 17, 2024
Link

Was COVID cooked up in a lab? Do “they” have a cure for cancer and are hiding it from the public? Do pharmaceutical companies make up diseases to market otherwise failed drugs? Did insurance companies scheme with infectious disease professional organizations to suppress treatments for chronic Lyme disease? Did the supplement industry lobby congress to weaken regulations of their own products?
OK, that last one is real, but the others are all examples of grand conspiracy theories involving health care. Conspiracy theories, of course, are not unique to health and medicine, they are common and on the rise in our society. Increasingly, however, health care professionals and organizations have to deal with the conspiracy phenomenon, which can directly impact both patient care and public health.
Conspiracy theories are also nothing new. They have certainly been amplified by social media – only minutes after the failed assassination attempt on Trump, various conspiracies were exploding on social media – but they long predate modern mass media.
What Are Conspiracy Theories?
Conspiracies, of course, are real and happen. Whenever two or more people get together and conspire to commit a crime, deception, or something nefarious, that is a conspiracy. But when we refer to a “conspiracy theory” that is short hand for a grand conspiracy, something involving massive scope in terms of people, time, and organizations.
The structure of a grand conspiracy theory generally involves one or more shadowy groups who have secret and expansive power. This group secretly uses their power to control some aspect of our world for their own benefit. The conspiracy often involves victims, which may be everyone. Typically the conspirators have incredible reach and power, in order to be able to pull off the conspiracy. And yet they also tend to make dumb mistakes, which is how the conspiracy theories can see the truth of what is happening. Everyone else are “sheeple” who are blind to the hidden truth of the conspiracy.
Why are conspiracy theories so compelling to so many? That is a question psychologists have been asking for decades, and they have a current partial answer. Research has shown that many people are simply opportunistic conspiracy theorists – they latch onto a single or a few conspiracies that seem to support their ideology, cultural tribe, or world-view. If, for example, you sell or promote some “alternative” medicine, the Big Pharma conspiracy theory is convenient. Why is your blatant pseudoscientific nonsense rejected by mainstream scientific organizations? There must be a conspiracy of silence and suppression.
A subset of the population, however, are all-purpose conspiracy theorists, endorsing many conspiracy theories across the ideological spectrum. These individuals tend to have more intuitive rather than analytical thinking styles, and some combination of enhanced pattern recognition and anemic reality testing.
Conspiracy thinking, however, is not unique to these predisposed individuals. Rather, conspiracy thinking seems to be a near universal programming flaw in the human brain. Humans are wired for “hyperactive pattern recognition” – we see patterns everywhere. But then our brains are supposed to filter out the patterns that don’t make sense. This can be a complicated process, however, and humans are only semi-rational beings, subject to a host of biases, flawed heuristics, and emotional influences. It only takes a bit of motivated reasoning, subjective validation, and confirmation bias to convince ourselves that even strange beliefs may be true.
Conspiracies are a special subset of false beliefs that are designed for self-reinforcement and insulation from refutation. Any bit of evidence for a conspiracy that is missing was suppressed by the conspirators. Any evidence that contradicts the conspiracy is a “false flag” operation. Anyone who refutes the conspiracy is obviously part of the conspiracy, or a paid shill. Once these logical barriers have been erected, there is no way out of the conspiracy theory. Then we can fully engage our pattern recognition powers to see connections everywhere, reinforcing the core conspiracy belief.
Another aspect of the grand conspiracy phenomenon is that they tend to enlarge over time, almost by necessity. Why hasn’t the media blown the whistle on 9/11, or the hidden cancer cure? Well, they must be in on the conspiracy. How could one pharmaceutical company control all the research that would underpin a cancer cure? All pharmaceutical companies must be working together, and they control all universities, all researchers, professional organizations, and regulatory agencies. And this conspiracy is international, involving every developed country in the world. Nothing short of that could possibly keep a lid on all of the scientific knowledge and research that would lead to something as powerful as a cure for cancer.
How do we defend ourselves from falling into the conspiracy trap?
First, be aware of the structure of grand conspiracy theories and look out for them. Don’t dismiss evidence as false simply because it contradicts your belief. Don’t dismiss the lack of evidence either.
Do not make “they” statements. The vague reference to “they” whitewashes many of the logical and empirical flaws of conspiracy theories. Who, exactly, are “they”, how do they operate, how could they pull off the alleged conspiracy? “They” is also used to conflate the actions of one person or one entity with a larger social group. It assumes collaboration and coordination. In one word, it basically assumes and avers a conspiracy.
A basic rule of critical thinking that applies beyond just conspiracy theories is to deliberately think of all the reasons why your belief is implausible or wrong. Don’t just look for support (we are good at making apparent connections and finding support for what we want to believe). Try to disprove yourself.
Grand conspiracy theories rapidly collapse under their own weight. The more you think about the logistics of the conspiracy, the bigger they have to become, until before long you are claiming there is a worldwide secrete government of alien reptiles from another dimension. Even without the alien reptiles, grand conspiracy theories are incredibly implausible.
Don’t fill in the blanks of your own ignorance about how a complex system works with simplistic conspiracy theories. This comes from the humility aspect of critical thinking. The world is incredibly complex, and no one has detailed knowledge of every profession, every institution, and every subculture. If you are not involved with the world of medicine, you may not understand how it works. It’s a giant black box. Don’t assume that anything can happen inside that black box, that it is monolithic, that everything that comes out of it is deliberate and nefarious. You can also ask actual experts to fill in some blanks, or if something you heard about online is plausible.
Finally, don’t spread conspiracy theories, online or elsewhere. Part of the appeal of conspiracy theories is that they can make people feel smart and clever, that they have seen the man behind the curtain, and are not among the “sheeple”. It also can give us a sense of understanding and therefore the illusion of control. Or a specific conspiracy theory can reinforce our existing belief systems and enhance our connections to our social group. This is a powerful motivation to want to spread these beliefs.
Don’t do it. Think about it first. Try to disprove your own theory. Check with those who may have more information.
I will also note that grand conspiracy theories historically have a terrible track record. So at least consider the probability that if you endorse a grand conspiracy, you are overwhelmingly likely to be wrong.

The Threat of Conspiracy Thinking to SBM
Conspiracy thinking is on the rise, and threatens the very notion of science and truth.
Steven Novella on January 20, 2021
Link
Science-Based Medicine (SBM) is, as the name plainly states, based upon and in fact dependent on science. So anything that threatens the institutions of science is a threat to SBM. Threats come in many forms – including pseudoscience (science distorted and biased to produce desired outcomes rather than true outcomes), anti-science (attempts at undermining or even replacing science as a method for figuring stuff out), and science-denial (perverting skepticism to deny established science) to name perhaps the top three.
But there is one thing that underlies and reinforces all three of these threats to science, and seems to be growing in power and influence: conspiracy thinking. As a psychological phenomenon this does not refer to just the notion that conspiracies exist, for they certainly do. Rather, it refers to belief in one or more grand conspiracies, a conspiracy so large that it would require hundreds or thousands of people working across institutions, nations, and perhaps even generations to create not just a lie but a false reality. Grand conspiracies are inherently implausible, as the larger they become the more quickly they would be revealed, eventually becoming mathematically impossible. As I like to say, grand conspiracies would tend to collapse in on themselves.
Grand conspiracies are nothing new, but the emergence of various popular and increasingly-implausible conspiracy theories calls out for explanation. The anti-vaccine movement, for example, has always been based on the big conspiracy, that the government (in fact many governments), the medical establishment, hospitals, universities, professional organizations, and pharmaceutical companies are all working together to promote the lie that vaccines are safe and effective. The anti-vaccine grand conspiracy, like any other, gets more implausible the more you think about it. The problem with conspiracy theories is that they need to get bigger and deeper in order to cover their tracks. In this way conspiracy thinking is a positive feedback loop, leading to grander and grander conspiracies.
For vaccines we could ask, why hasn’t any intrepid reporter broken this story? Because they must be in on it. Further, the conspiracy must simultaneously be massive and airtight. The medical establishment must have engineered the illusion that vaccines have significantly reduced many diseases, and then tricked millions of healthcare experts or involved them in the conspiracy. The conspiracy is so far-reaching that it has controlled every single government around the world. None have broken away from its grip.
The grand conspiracy narrative also relies upon an implausible and cynical view of humanity. People are divided into three stark groups:
    • The conspirators, who are simultaneously brilliant and absurdly stupid since they can engineer an unbelievably complex conspiracy but always reveal themselves in silly ways.
    • The enlightened, members of the army of light who sees the conspiracy for what it is and are on a mission to save the world.
    • Everyone else, the “sheeple” who cannot see the conspiracy and remain its victims.
The psychological appeal of being among the enlightened few is clear.
Conspiracy thinking is anathema to science, because it turns the rules of evidence on its head. In the world of grand conspiracies a lack of evidence is simply evidence that there has been a cover-up – no evidence is evidence for the conspiracy. Any positive evidence that contradicts the conspiracy is also more evidence for the conspiracy, because it was planted by the conspirators. Therefore, there is no way to refute the conspiracy. It is immune to evidence which is inherently anti-science.
Even those, like myself, who have taken an interest in the grand conspiracy phenomenon for decades have been surprised by the level they have reached in the last couple of decades. Just when you think that you have seen the pinnacle of conspiracy absurdity, new ground is broken. I used to think that the entire alternative medicine grand conspiracy narrative was as bad as it gets. In this narrative, not only vaccines, but virtually all of modern medicine is a corporate conspiracy. Anyone who defends any science-based practice must be in on the conspiracy, and suppressing the “truth” about natural remedies. (Yes, I still get these e-mails.) So askew of reality is conspiracy thinking that it claims millions of people who have essentially dedicated their lives to the healing profession are hiding the cure for cancer, knowingly injecting poison into children, and laughing all the way to the bank. The CAM conspiracy theory requires an entire profession and their enablers to be guilty of psychopathic evil.
As absurd as all this is, and as jaded as I have become about the limitlessness of human gullibility, I was surprised by the popularity of the Flat-Earth conspiracy. This is the notion that the entire scientific community, along with the aerospace and commercial airline industries, not to mention everyone with a telescope, is engaged in a multi-century conspiracy to convince the world that the Earth is roughly a sphere when in fact it is flat. This is an excellent example of how conspiracies need to become exponentially more absurd as you challenge their premises. If the Earth is flat, then it would collapse under its own gravity. So therefore – gravity must also be a conspiracy. That’s right, gravity.
The Flat-Earth conspiracy is so grand, in fact, that it functions as an ultimate conspiracy. An ultimate conspiracy is one so large that it essentially breaks truth. If the world can be lying about something so fundamental as the shape of the planet, then you cannot believe anything. “They” can be lying about anything and everything, there is no truth, there are no facts, and you can then justify every other conspiracy because no matter how grand they are, they pale in comparison.
It was not surprising, therefore, that the Flat-Earth conspiracy paved the way for Q-Anon – which started as the claim that the Democratic party is made of devil-worshipping pedophiles and that Trump (any day now – better hurry) is going to expose them and bring them to justice. Q-Anon started with this core claim but has metastasized into an all-purpose right-wing grand conspiracy, including anti-vaccine views.
Trying to reverse engineer how conspiracy theories have gotten so out of control in the last two decades should keep psychologists busy for a while. Social media, however, seems to be lurking at the center of this phenomenon. Social media algorithms have created another positive feedback loop, sucking in otherwise typical and even intelligent people into a conspiracy narrative until they cannot tell fact from fiction, and then they can believe anything. This phenomenon is nothing less than an existential threat, including to science and therefore SBM. But more broadly this is a threat to any notion of evidence, logic, facts, and reality. We need to take this threat seriously, and devote time and attention to figuring it out and mitigating it.

All science denial is a form of conspiracy theory
Regular readers of this blog know that many forms of quackery and science denial have conspiracy theories associated with them, but a further examination suggests that all forms of science denial are a form of conspiracy theory. In the middle of a deadly pandemic, science denial represents a form of conspiracy theory with potentially deadly consequences.
David Gorski on January 25, 2021
Link

Last week, SBM’s fearless founder Steve Novella wrote what I considered to be an important post about the danger of conspiratorial thinking to science-based medicine (SBM), noting that anything that threatens the institutions of science, such as conspiratorial thinking, is a huge threat to science. He correctly noted one example of pseudoscience that is based on conspiratorial thinking, namely the antivaccine movement. Indeed, I once noted that all antivaccine views—and, no, I’m not going to qualify that statement, as I do mean all antivaccine views—are ultimately based on, or, in the case of the vaccine-hesitant at least supported by, a grand conspiracy theory that six years ago I dubbed “the central conspiracy theory of the antivaccine movement“.
What is that conspiracy theory? Regular readers will recognize it immediately when I characterize it. In brief, the central conspiracy theory of the antivaccine movement is that vaccines are dangerous (and, in many tellings, ineffective) but that “they” are hiding the evidence of that harm and ineffectiveness. Who are “they”? Obviously “they” include the CDC, the FDA, big pharma (of course!), the medical profession, the press, and pretty much everyone else outside of the select, small group of those who are enlightened and thus privy to this “hidden knowledge”. Although it seemed odd at the time to many, in retrospect it shouldn’t have been (and wasn’t) so strange how soon after the COVID-19 pandemic hit antivaxxers made common cause with COVID-19 deniers. Of course, I’ve discussed the importance of conspiracy theories in medical quackery, especially the antivaccine movement and COVID-19 denial, several times before. This time, however, I’d like to broaden the discussion.
Obviously, I agree with Steve regarding the danger of conspiratorial thinking to SBM. Where we differ (and some might even view it as quibbling, more of a different in emphasis rather than substance) is that, to me, Steve doesn’t go far enough. If there’s anything that the pandemic has taught me, with the help of Mark Hoofnagle, it’s that all science denial is rooted in conspiracy theory. Steve mentioned, for instance, flat earthers and QAnon. QAnon, of course, is basically the ur-conspiracy theory for the age of Donald Trump, a conspiracy theory so adaptable that it can be all things to all people, even as others have pointed out that Q is very much akin to the old Jewish Blood Libel conspiracy theory, rebranded and revamped for the Facebook century, with more than a dash of the Satanic panic of the latter decades of the last century. Remember, at its heart, the QAnon conspiracy theory claims that there is a secret cabal of Satan worshipers (who are also pedophiles) who secretly rule the world behind the scenes from positions of power in the government, banks, news media, entertainment industry, and church. (Oh, and they’re also cannibals, killing children for the adrenochrome in their blood.)
In addition to the antivaccine movement, Steve also mentioned QAnon (of course!), alternative medicine supporters, and the flat earth movement. Now let me readjust the emphasis and introduce what I would like to refer to as the central conspiracy theory of science denial. It’s basically the central conspiracy theory of the antivaccine movement, just much broader.
The central conspiracy theory of science denial
I begin this section by restating the central conspiracy theory of the antivaccine movement: Vaccines are harmful (and don’t work), but “they” are hiding the evidence of this. Now, let’s take the title of a book that was among the things that got me interested in investigating the claims and appeal of alternative medicine: Kevin Trudeau’s book, Natural Cures “They” Don’t Want You to Know About, originally published in 2005, which, unsurprisingly, was the year when I first started blogging in earnest on my first blog. The central premise of the book was, of course, that there are “all-natural” cures for basically all illnesses, be they serious or less so, including cancer, herpes, arthritis, AIDS, acid reflux disease, phobias, depression, obesity, diabetes, multiple sclerosis, lupus, chronic fatigue syndrome, attention deficit disorder, muscular dystrophy, etc., but that these “cures” are being intentionally “hidden” and “suppressed.” And who’s “hiding” and “suppressing” these “cures”? It is, of course, the usual suspects: The FDA, the Federal Trade Commission (FTC), the CDC (in the case of vaccines and “natural cures” for vaccine-preventable diseases), the major food and drug companies, and, of course, the entire medical profession, all in an effort to protect the profits of these industries and the authority of governmental agencies and the medical profession.
Now let’s look at a sampling of various forms of science denial and the conspiracy theories associated with each of them:
    • Evolution denial (a.k.a.) creationism. Creationists deny the science of evolution. Among creationists, there are two major types: “Young earth” creationism (YEC) and “intelligent design” creationists. YEC completely deny evolution in favor of a strict literalist interpretation of the Bible in which all lifeforms were created by God roughly 6,000 years ago and all the science that says the earth is billions of years old is false. “Intelligent design” creationism (ID) concedes that evolution has occurred (to some degree or other) and that the earth might be 4 billion years old, but denies that evolution by natural selection (and other forms of selection) is the driving force that resulted in the current diversity of life that we now observe. Instead, ID posits that there was an “intelligent designer” who guided (and continues to guide) evolution. ID creationists like to obfuscate who this “designer” might be, but, however much they try to obfuscate, it’s clear that the “designer” is God and that ID is, at its heart, little different from YEC other than in sophistication. Personally, I like to liken ID creationists to antivaxxers who claim that they’re “not antivaccine” but rather “vaccine safety advocates” in that, like such antivaxxers are trying to hide that they are antivaccine, ID creationists try to hide that they are anti-evolution. In this model, as is the case for antivaxxers who proudly proclaim that they are antivax, I almost have more respect for young earth creationists, because they at least are being more honest that they are antievolution and that their resistance to evolution is based on a literal interpretation of the Bible. In any event, the central conspiracy theory of creationism, be it YEC or ID, is that a shadowy cabal of atheists, biologists (a.k.a. “Darwinists” in creationist parlance), and “secularists” are “hiding” or “suppressing” the evidence that “Darwinism” is a “sham”. Usually, the motivation of these “atheists” and “Darwinists” is to deny and suppress religion.
    • Climate science denial. Currently, the scientific consensus regarding climate change is that the earth is warming (and has been since the industrial age started) due to human activity, specifically the CO2 and other “greenhouse gases” released into the atmosphere primarily by the burning of fossil fuels, leading to the “greenhouse” effect trapping heat in the atmosphere. More recent evidence suggests the possibility, even likelihood, of catastrophic increases in average global temperatures over the next several decades that will result in, among other things, the melting of glaciers and polar ice (with resulting elevations in sea level), more extensive droughts, more extreme weather, ocean disruption, and worse. Climate science thus concludes that the only way to avert potential catastrophe is to massively (and rapidly) decrease the burning of fossil fuels. Climate science denialists, as you might expect, deny the climate science that concludes that human activity is the primary driver of climate change and overall global warming. Their central conspiracy theory is that a cabal of “radical environmentalists,” governments, universities, and climate scientists are “hiding” and “suppressing” the “real” data showing that human activity is not causing global warming and/or “manipulating” the data to make it appear that human activity is causing increased CO2 levels and warming the planet.
    • Alternative cancer cures. Believers in alternative cancer cures claim that there are, as Kevin Trudeau claimed, “natural cures” for cancer, although they do not limit themselves to just “natural cures.” (After all, Stanislaw Burzynski’s “antineoplastons” are hardly “natural” when you come right down to it, even though they were sold as “natural” endogenous cancer fighting compounds and originally isolated from human urine; on second thought, I guess that’s “natural” enough.) The central conspiracy theory here is—you guessed it—that the FDA, oncologists, scientists, and—of course!— big pharma are “suppressing”/”denying”/”covering up” the evidence that these cancer cures exist and work, all to protect the profits of big pharma and the power of government regulatory agencies like the FDA. In fact, alternative medicine in general denies the efficacy of modern medicine and claims that there are “natural” cures being hidden from you or that various conditions we treat to prevent disease, such as hypercholesterolemia, are not really a risk to your health.
    • COVID-19 denial. COVID-19 deniers claim that COVID-19 is in actuality not a serious illness, that the pandemic is overhyped, and/or that it was intentional. I’ve written about the conspiracy theories that flow from COVID-19 denial before on multiple occasions. Most prominent are conspiracy theories that claim that the pandemic was really a “plandemic” intentionally caused by a shadowy cabal of governments, the World Health Organization, scientists, and (of course) Bill Gates, all in order to impose authoritarian controls on the world. (One such conspiracy theory, the Oblivion Agenda, even claims that COVID-19 is a bioweapon introduced by an unholy alliance of the global elite and aliens to “depopulate” the world in order to allow the elites to let the aliens in and thereby profit. I kid you not.) The other version of this conspiracy theory is the “casedemic,” which claims that the pandemic is a result of PCR tests for SARS-CoV-2, the coronavirus that causes COVID-19, in which the cycle threshold is set too high and thus too sensitive, producing way more false positives than true positives. The motivation? Again, control and profit on the part of drug companies, testing companies, and the government. How to explain the >400,000 deaths in the US (and climbing rapidly)? COVID-19 deniers claim that the vast majority of these deaths were really due to something else and intentionally misattributed to COVID-19.
    • Anti-GMO. Anti-GMO activists demonize genetically modified organisms based o the same techniques common to science denial: cherrypicked studies, misrepresentation of studies, confusing correlation with causation, pseudoscience, logical fallacies, fake experts, and more. Then, of course, there is the conspiracy theory behind it all, that companies that sell GMOs, especially Monsanto, are “suppressing” the “truth” about GMOs (namely that they are harmful to your health), all in order to profit and cement control over agriculture. Of course, it doesn’t help that Monsanto has engaged in questionable business practices; that history makes it very easy for anti-GMO conspiracy theorists to use Monsanto’s past actions to imply a “coverup” or that the science really does show that GMOs are harmful. (They aren’t.)
    • Germ theory denial. It’s been a depressingly short time since I’ve discussed germ theory denial, namely the claim that that microbes not the primary cause of infectious disease. As hard as it is to believe, germ theory denial exists in the age of the pandemic, with the usual claim being that it is the “terrain” (i.e., the health of the body) that determines whether one gets sick, not the microbe. While it is true that there is a germ of truth in that claim, germ theory deniers take it farther, to the point of arguing that you can somehow “turbocharge” your immune system “naturally” in order to make yourself, in essence, immune to any infectious disease. Naturally, physicians and big pharma don’t want you to know this.
    • Antivaccine pseudoscience. Coming back to the antivaccine movement, again, the central conspiracy theory of the antivaccine movement is that vaccines are harmful/don’t work but that “they” are hiding/covering up/manipulating the evidence that demonstrates vaccines’ harm and ineffectiveness. Again, it’s the usual cast of characters, the CDC, FDA, federal and state governments, the medical profession, and (again, of course!) big pharma who are behind the conspiracy. Indeed, there have been two major variants of the central conspiracy theory of the antivaccine movement that I’ve known since I started paying attention. The first was the Simpsonwood conspiracy theory and was promulgated by Robert F. Kennedy, Jr. in 2005; the second was the “CDC whistleblower” conspiracy theory and was promulgated by Brian Hooker and Andrew Wakefield in 2014, later amplified by Andrew Wakefield and Del Bigtree in the antivaccine propaganda movie VAXXED. Both conspiracy theories claim that the CDC “manipulated” the data” in order to “hide” or “suppress” the evidence that vaccines cause autism. And don’t even get me started on the antivaccine Bill Gates conspiracy theories.
Lest supporters of so-called “complementary and alternative medicine” (CAM), more recently rebranded as “integrative medicine” or “integrative health” think that they are immune and that many of their beliefs aren’t rooted in conspiracy theories, just take a look at some of the leading lights of the movement, such as Deepak Chopra or Mark Hyman, both of whom have engaged in conspiratorial thinking and embraced forms of pseudoscience rooted in conspiracy theories. For example, Deepak Chopra has long engaged in a form of evolution denial in which he denied that genes are deterministic that I used to write about regularly 14 years ago, the better to support his idea that the universe has “consciousness” and “purpose.” Mark Hyman, of course, co-authored an antivaccine book with Robert F. Kennedy, Jr., and that didn’t stop the Cleveland Clinic from hiring him to start a “functional medicine” clinic.
One can also look at other forms of denial not related to science and see the same characteristic of having a conspiracy theory at the root. My favorite example is Holocaust denial, about which I used to write extensively back in the day. The conspiracy theory behind Holocaust denial is, of course, that the Jews (and often Communists, whom antisemites often view as more or less synonymous with Jews as having been created by “international Jewry”), who suppress/falsify/manipulate the evidence showing the Holocaust didn’t happen/killed many times fewer than 6 million Jews for their own nefarious purposes, namely control, power, and money. (Holocaust deniers even refer to the “Holocaust industry” in much the same way that quacks and antivaxxers refer to big pharma.) Unfortunately for Holocaust deniers, their motivation is very transparent. As Holocaust historian Deborah Lipstadt once described, “the real purpose of Holocaust revisionism is to make National Socialism an acceptable political alternative again”.
Of course, then there are other forms of denial, such as the aforementioned flat earth movement, as discussed by Steve last week:
As absurd as all this is, and as jaded as I have become about the limitlessness of human gullibility, I was surprised by the popularity of the Flat-Earth conspiracy. This is the notion that the entire scientific community, along with the aerospace and commercial airline industries, not to mention everyone with a telescope, is engaged in a multi-century conspiracy to convince the world that the Earth is roughly a sphere when in fact it is flat. This is an excellent example of how conspiracies need to become exponentially more absurd as you challenge their premises. If the Earth is flat, then it would collapse under its own gravity. So therefore – gravity must also be a conspiracy. That’s right, gravity.
The Flat-Earth conspiracy is so grand, in fact, that it functions as an ultimate conspiracy. An ultimate conspiracy is one so large that it essentially breaks truth. If the world can be lying about something so fundamental as the shape of the planet, then you cannot believe anything. “They” can be lying about anything and everything, there is no truth, there are no facts, and you can then justify every other conspiracy because no matter how grand they are, they pale in comparison.
But why do conspiracy theories and science denial fit together so well, hand-in-glove? Why do I argue that all science denial is conspiracy theory, when you strip away the camouflage and reveal its core? The reason is simple. Science denial, whatever the motivation for the denial, requires conspiracy theory, because of the very characteristic that Steve cites above.
Why is the central conspiracy theory necessary for science denial?
Let’s look at some examples of science denial and ask: Why is a conspiracy necessary for each of them? It is not for nothing that evolution is considered the central theory of biology, so important is it to our understanding of life. Let’s say that you “question” evolutionary theory, that you don’t want to believe that evolution, by natural selection and other forms of selection, is the central driving force that produced the diversity of life. How, then, do you explain the fact that, a completely negligible minority aside, biologists overwhelmingly support the theory of evolution as the central organizing principle of their scientific discipline and generally don’t argue whether evolution by natural selection is critical, but rather how critical? Let’s look next at climate science. You doubt climate science and refuse to believe that humans are primarily responsible for the no-longer-gradual increase in global temperature over the last several decades? How do you explain the fact that, a completely negligible minority aside, climate scientists overwhelmingly support the hypothesis that human activity is the primary driver of global warming and generally argue not about whether human activity is driving global climate change but rather about how much and if it’s still possible to slow or stop the change in climate in time to avert catastrophe? Let’s circle back again to antivaccine pseudoscience. If you believe that vaccines cause autism, sudden infant death syndrome, infertility, alterations in your DNA, and even the death of teenaged girls (Gardasil, of course), how do you explain the fact that, a completely negligible minority aside, scientists and physicians overwhelmingly have concluded that vaccines are safe and effective and do not cause any of those problems?
It takes a conspiracy theory, of course, to explain why experts so overwhelmingly reject your worldview, because, surely, it can’t be because you’re just plain wrong, right? Surely, the reason why nearly all the relevant experts in the relevant scientific field reject your viewpoint and beliefs and the evidence reported in the scientific literature in that field overwhelmingly rejects—or, at minimum, emphatically does not support—your beliefs is because there’s a vast conspiracy to suppress them and they are in on it. Again, surely it can’t be because you’re just plain wrong, can it?
Of course not.
Then, of course, besides an explanation for why science rejects your viewpoint, conspiracy theories make you the hero. Not only do you and your fellow conspiracy theorists possess “hidden knowledge” that the rest of the world does not have (or foolishly rejects), but you are the hero fighting against a vast and evil cabal seeking to suppress that hidden knowledge. I like to point to the example of someone like antivaccine activist Kent Heckenlively. As I’ve put it before, every story must have a victim, a hero, and a villain; a conspiracy theory lets someone like Mr. Heckenlively portray himself as the hero fighting the villain (big pharma, the government, etc.) for the “victims” (the “vaccine-injured” children). Indeed, Mr. Heckenlively even imagined himself Aragorn, son of Arathorn, in The Lord of the Rings, marching to the Black Gate of Mordor on a doomed mission that he didn’t expect to survive, all in order to distract the Dark Lord Sauron, so that Frodo and Sam could complete their mission. You see the same sort of fantasy in QAnon believers, who think themselves heroes “protecting the children”. It’s a powerful combination, an explanation for why your views are rejected by science and the ability to paint yourself as a hero seeking to bring to light hidden knowledge and bring down a great evil.
It’s been argued that science denial has five characteristics:
     1.  Conspiracy theories
     2.  Fake experts
     3.  Cherry picking
     4.  Impossible expectations of what science can do or offer
     5.  Misrepresentation and logical fallacies
More and more, I’m coming to the conclusion that the last four characteristics all flow from the first and that science denial is, at its heart, a conspiracy theory when you strip away all the other characteristics. I don’t claim to be the originator of this idea, either. I blame Mark Hoofnagle for influencing me. Of course, conspiracy theories are defined by their own characteristics, all of which apply to science denial as well.
True believers vs. grifters
Of course, even if you accept that science denial is a form of conspiracy theory, one must accept that there are…complexities. After all, there are real conspiracies in the world. The difference between conspiracy theories and real conspiracies is that real conspiracies are not, like conspiracy theories, unfalsifiable and ever-evolving in order to remain so. Real conspiracies can be discovered and proven through standard investigational techniques used by law enforcement and journalists the world over—and have been. For example, there really was a conspiracy to bring down the World Trade Center and Pentagon by flying commercial jetliners into them in 2001; it just wasn’t the Mossad and the US government who were behind the conspiracy, as “9/11 Truth” conspiracy theorists would have you believe. The petroleum industry did conspire to cast doubt upon the climate science. Big Tobacco did conspire to deny, obfuscate, and suppress the scientific evidence linking smoking tobacco products to lung cancer and other disease. Again, some conspiracies are real. Again, the difference between a conspiracy theory and a real conspiracy is that conspiracy theories are virtually unfalsifiable and real conspiracies can be uncovered and demonstrated by standard investigative techniques.
These examples also bring me naturally to grifters.
Wherever you find science denial, almost inevitably you will also find grift. The antivaccine movement is a great example, with a veritable panoply of pure grifters ranging from Andrew Wakefield to Joseph Mercola to Del Bigtree to Robert F. Kennedy, Jr. to Dr. Paul Thomas to so many, many others that I really can’t name them all. The denial of oncology brings to mind all sorts of grifters selling alternative cancer cures, Stanislaw Burzynski being the most prominent one who comes to mind. Homeopaths, naturopaths, chiropractors, and other quacks who deny SBM very frequently have something to sell, usually supplements or some other “medical treatment” not based in science. Thus, feeding antiscience conspiracy theories is a strategy by which grifters hawk their wares.
Similarly, those selling an ideology also take advantage of antiscience conspiracy theories, and vice-versa. In other words, many conspiracy theories are tactical – the origin of some conspiracy theories is not genuinely held erroneous beliefs, but rather they are the result of an intentional campaign of disinformation designed to produce a political or ideological end. For example, fundamentalist religious activists who view the science of evolution as a threat to their worldview and their faith, willingly stoke conspiracy theories of evolution denial. Conservative free market fundamentalists, who abhor anything that would justify a larger role for government or increased government regulation, are more than happy to spread the conspiracy theory behind climate science denial. True believers in “natural” remedies are more than happy to spread conspiracy theories about the government, big pharma, and physicians “suppressing” alternative medicine. Sometimes the converse is true, as well, with science-denying conspiracy theorists using ideology as a gateway through which those holding that ideology can be brought into the conspiracy theory. The most obvious example to which I like to point is how the antivaccine movement so skillfully co-opted conservative rhetoric of “health freedom”, “parental rights”, and hostility towards government mandates and regulations to attract conservatives to their opposition to vaccine mandates. Unfortunately it’s worked spectacularly well, leading in a political shift in the antivaccine movement to the right over the last several years and a disturbingly large number of Republican politicians pandering to antivaxxers or even being antivaccine themselves. Meanwhile, antivaxxers have been actively coordinating their activities and synchronizing their key messages in order to cast doubt on COVID-19 vaccines.
Science denial is a conspiracy theory
As Steve noted last week, science denial is not just a threat to SBM, but to all science and to, as he put it, “any notion of evidence, logic, facts, and reality”. The same is true of conspiracy theories. As I contemplated this post, I asked the proverbial “chicken or the egg” question: Which came first, conspiracy theories or science denial? Most likely it was conspiracy theories, which date back as far as recorded history can take us and very likely predate science. Whatever the answer, though, more and more I’m coming to the conclusion that science denial is a form of conspiracy theory and that we will not successfully mitigate science denial until we are able to understand and mitigate conspiracy theories because, even if you do not accept the proposition that all science denial is a form of conspiracy theory, it is without a doubt true that all science denial relies at least in part on conspiracy theories to support it. As we’ve seen in the COVID-19 pandemic, the conspiracy theory of science denial can have deadly consequences, consequences that, once the pandemic finally abates, will become apparent as the earth’s climate continues to warm due to human activity. Worse, as Steve also discussed, science denial has been turbocharged by social media in an unprecedented manner. Misinformation and conspiracy theories travel farther and permeate the consciousness of more people than has ever been possible for them to do before. Developing strategies to combat this tendency and bring people back to reality is arguably the existential problem of the 21st century.

Frozen meat vs. COVID-19 misinformation: The strange case of Steak-umm
Of all the strange things that have happened during the COVID-19 pandemic, the rise of a frozen meat company as a source of critical thinking and skepticism regarding COVID-19 misinformation and disinformation is one of the strangest. How did Steak-umm become a champion of critical thinking about the pandemic?
David Gorski on November 16, 2020
Link
Since the COVID-19 pandemic hit the US close to nine months ago, a number of bizarre things that I never thought I’d see have happened. For example, I never thought I’d see a President of the United States promote unproven treatments for a deadly viral illness, as President Trump did with hydroxychloroquine. (Dr. Mehmet Oz and other medical grifters promoting unproven treatments, sure, but the President of the United States? Well, I guess I did forget momentarily who our President was.) Nor did I ever think I’d see a President say such incredibly stupid things about potential treatments for such a disease, including using light and disinfectants internally to fight the coronavirus. Don’t get me wrong, though. Given my background, certainly, I did expect there to be COVID-19 conspiracy theories and wasn’t particularly surprised to see the antivaccine movement team up so quickly with COVID-19 conspiracy theorists and cranks given their shared world view viewing the same groups as villains hiding “The Truth” from you—yes, you!—but even I was a bit taken aback at how vociferously antimask ideology became a thing even though masks do work to slow the spread of COVID-19.
Perhaps the strangest thing, though, that happened during the pandemic is the rise of the most unexpected source of reason, science, and information to fight disinformation. I’m referring to the Twitter feed of a frozen meat company called Steak-umm. To give you an idea what I mean, here’s the Twitter thread pinned to the top of Steak-umm’s Twitter profile:
friendly reminder in times of uncertainty and misinformation: anecdotes are not data. (good) data is carefully measured and collected information based on a range of subject-dependent factors, including, but not limited to, controlled variables, meta-analysis, and randomization
— Steak-umm (@steak_umm) April 7, 2020
outliers attempting to counter global consensus around this pandemic with amateur reporting or unverified sourcing are not collecting data. breaking news stories that only relay initial findings of an event are not collecting data. we have to be careful in our media consumption
— Steak-umm (@steak_umm) April 7, 2020
it can be difficult to know what to believe in a time when institutional trust is diminished and the gatekeepers of information have been dismantled, but it's more crucial now than ever before to follow a range of credentialed sources for both breaking news and data collection
— Steak-umm (@steak_umm) April 7, 2020
all we currently have are limited and evolving metrics that experts are deciphering and acting upon immediately to the best of their ability. this terrain leaves many openings for opportunists and charismatic manipulators to lead people astray by exploiting what they want to hear
— Steak-umm (@steak_umm) April 7, 2020
breaking news and storytelling will always be spun with interpretive bias from different media perspectives, but data is a science that can't be replaced by one-off anecdotes. try to remember this to avoid fear-based sensationalism or conspiracy theories taking over your mind
— Steak-umm (@steak_umm) April 7, 2020
you can maintain independent, critical thinking toward institutions without dipping into fringe conspiracies that get jumpstarted by individual anecdotes being virally spread as data. it's not easy, but it's necessary to keep any semblance of responsible online information flow
— Steak-umm (@steak_umm) April 7, 2020
we're a frozen meat brand posting ads inevitably made to misdirect people and generate sales, so this is peak irony, but hey we live in a society so please make informed decisions to the best of your ability and don't let anecdotes dictate your worldview ok
steak-umm bless
— Steak-umm (@steak_umm) April 7, 2020
I love how whoever’s running the Steak-umm Twitter feed frequently concludes the company’s Tweets with “steak-umm bless”. In any event, notice the date on this Twitter thread: April 6, 2020. This was early in the pandemic. Notice how what Steak-umm is saying could easily be something found in a Science-Based Medicine post about how to evaluate medical information, science, and misinformation. Notice the incredible level of self-awareness in the Tweet pointing out how Steak-umm knows its Twitter feed’s purpose is to sell its product and its ads might misdirect to generate sales, while pivoting to urge people to “make informed decisions to the best of your ability and don’t let anecdotes dictate your worldview”.
Certainly, I didn’t fail to notice the irony of finding myself, as someone who’s dedicated much of his life and career to promoting science-based medicine (SBM) and combatting medical misinformation and disinformation, be it cancer quackery, fear mongering about genetically modified organisms, antivaccine pseudoscience, or other medical conspiracy theories, all while, when I deem it appropriate and imperative, also taking on the poor evidence base for a number of “conventional” medical treatments, helping to amplify the message promoted by a frozen meat company on its Twitter feed, just because the message jibes with the message I’m trying to promote. Oh, well, if this post helps Steak-umm sell more frozen meat, so be it. Even so, in the interests of full disclosure, several weeks ago Steak-umm sent me some Steak-umm swag (a hat, some coupons, a facemask) because of my interactions on Twitter, but that was only after many weeks of positive interactions and with the explicit statement that I was under no obligation to write about Steak-umm. No doubt some quack or antivaxxer will accuse me of being a shill for big frozen meat. Such is the strangeness of the times we live in.
Ever since I first noticed Steak-umm’s rather…unusual…approach to brand promotion through Tweeting messages encouraging skepticism and critical thinking, particularly about the hysteria and dubious claims being made about the COVID-19 pandemic, several months ago, I was always a bit curious about how the company’s social media arm decided on this approach and whether it was successful. Certainly, it struck me as successful from the standpoint of promoting critical thinking about COVID-19, but how could such an approach also be effective advertising? Sure, Steak-umm got lots of free publicity, and, sure, it now has nearly 170K Twitter followers, but how was it used to sell frozen meat? In other words, how did Steak-umm do well by doing good?
I can’t believe I missed this article the first time it came around, but Ekaterina Bogomoletc at North Carolina State University and Nicole Lee at Arizona State University looked at this question in an article in the Journal of Business and Technical Communication entitled “Frozen Meat Against COVID-19 Misinformation: An Analysis of Steak-Umm and Positive Expectancy Violations“. Reading the article also let me learn a bit about expectancy violation theory.
Basically, expectancy violation is a theory in communications that tries to explain the behavior of human beings while interacting. This article sums it up rather succinctly:
Expectancy violation theory emphasizes on an individual perception of the interaction in a particular situation. People while communicating will create an expectation of how the other will react. Violation to this expectation can cause to [sic] a perception that will be positive or negative. People behave differently according to the cultural values they grow up in and this influences the reaction of the people considerably.
Expectancy violation theory also is dependent on the personal space. The personal space is the boundary we keep and freedom is given to the people whom we are close with. Particular personal space is expected from the people whom they interact with according to the relationship they have with them. The theory explains that people tends [sic] to protect the personal space when they experience a violation in the expected behaviour.
You can see how Steak-umm’s Twitter feed would violate people’s expectations, albeit in a positive manner. We expect corporate social media to be primarily advertising, nothing more. However, in this case, we see a company’s social media arm being used to promote critical thinking and science about a deadly pandemic, which is definitely not something I would have expected to see.
Bogomoletc and Lee observe about the thread above that I quoted:
This thread received thousands of reposts and earned Steak-umm the titles “voice of reason” (Masnik, 2020) and “beacon of truth” (Zafarris, 2020) in the media. Within the next week, Steak-umm thanked its followers for amplifying its messages and shared five more threads addressing science communication, misinformation, and critical thinking. Meanwhile, the company demonstrated self-awareness by acknowledging its bizarre new role, apologizing for anthropomorphizing the brand and admitting that its ultimate goal was to sell its products (Masnik, 2020).
Here are examples of Steak-umm’s self-awareness:
we are just a frozen meat brand tweeting into the void hoping to provoke thought, inspire unity, and relay useful information. we are not an expert in anything or some beef god. we are a company with a bottom line being run by people. stay safe and be good to each other out there
— Steak-umm (@steak_umm) April 10, 2020
And:
people think it's bizarre, ironic, and funny when a frozen meat company points out the importance of critical thinking, but chances are the same message would never "go viral" if it was from a person. our society values entertainment over truth and that's a huge problem
— Steak-umm (@steak_umm) April 8, 2020
uncertainty doesn't sell. nuance doesn't sell. long, complex lectures don't sell. a video clip of someone saying "it's complicated" will never perform the way one would of someone using confident, flippant, polarizing rhetoric, and that's a huge problem
— Steak-umm (@steak_umm) April 8, 2020
getting people credible, thoughtful, and impartial information is an uphill battle because most have evolved (and been programmed) to not care unless it's entertaining and tells them what they want to hear, but we have to try in these times of cultural crisis and polarization
— Steak-umm (@steak_umm) April 8, 2020
And, there was another thread:
one major roadblock in getting credible information to the masses is the divide between experts and communicators. most experts aren't communicators and most communicators aren't experts. this often results in research being spun with a narrative by the time it reaches the public
— Steak-umm (@steak_umm) April 9, 2020
research makes descriptive claims, while humans draw prescriptive claims from it. these interpretations are often necessary to discover the "why" and "how," but one fact remains: the experts who do the work itself are the ones with the best interpretation of their own methodology
— Steak-umm (@steak_umm) April 9, 2020
since experts are often inadequate communicators, this allows space for anyone with a microphone to interpret their research however they want and project it to the masses, even if it directly opposes their conclusions. this drives partisan divide, as well as fringe conspiracies
— Steak-umm (@steak_umm) April 9, 2020
there are cases when experts oppose the conclusions of other experts within their specific fields. that debate and refining is part of the scientific process, but just because someone is an expert in (x) doesn't mean they're an expert in (y), and each expert's credentials vary
— Steak-umm (@steak_umm) April 9, 2020
many people seek out fringe dissenters of an expert consensus to reinforce their biases, and many times those dissenters are not credentialed in the relevant area, meaning they are:
1) wrong2) wrong / acting in bad faith3) wrong / a useful idiot4) the next galileo (unlikely)
— Steak-umm (@steak_umm) April 9, 2020
the vast majority of any population look to communicators (often entertainers) for answers. because of this, we need to demand higher standards of our communicators when dealing with expert information, and we also need to encourage our experts to become better communicators
— Steak-umm (@steak_umm) April 9, 2020
if people can be educated on the motivated reasoning behind where they get information from, while remaining open to expert consensus, there's room for thoughtful discussion that will hopefully bend toward the truth. but it has to start with genuine understanding
steak-umm bless
— Steak-umm (@steak_umm) April 9, 2020
Again, there’s the “steak-umm bless,” which is often how Steak-umm signals its readers that a thread is finished. Steak-umm has also been raising money for Feeding America, to which the brand donated $25,000 in April.
These are just a few examples from over seven months ago. Here’s one from Friday:
how do you determine if a source is “credible?” well, it's an incredibly complex question. you need to understand the outlet and author's track record, their editorial process, associations, funding, audiences (market), critiques, and the influence of their work, for starters
— Steak-umm (@steak_umm) November 13, 2020
it’s important to determine what *kind* of source you’re sharing. is it an opinion piece? breaking news? substantiated reporting? an anecdote? is it a new study with little support? is it a statistic without context? a peer-reviewed study? based on expert consensus or outliers?
— Steak-umm (@steak_umm) November 13, 2020
there are various tools to use as a jumping off point for determining if a source is trustworthy, such as allsides, media bias fact check, snopes, and so on. however, these tools are limited in how they measure bias and still come from a POV, so you need to unpack more from there
— Steak-umm (@steak_umm) November 13, 2020
you can use reverse google image searches to find the origins of memes, you can customize search result dates in google, you can use google scholar to find studies (although you need to be trained in how to navigate and read studies before just searching for what confirms biases)
— Steak-umm (@steak_umm) November 13, 2020
trust in experts and institutions is at an all time low so it’s important for sources to be transparent about any biases and points of view they have. the standard for credible sources isn't objectivity, it's impartiality and accuracy through rigorous processes
steak-umm bless
— Steak-umm (@steak_umm) November 13, 2020
Which led to this response, among others:
Why is the frozen meat spokesperson one of the greatest voices of reason in America right now, while our “leaders” are spewing nonsense? It looks like the fall of the Roman Empire.
— Balgor 🇺🇦 (@Balgor11) November 13, 2020
I was basically asking the same question.
The authors ask a rather obvious question, noting first that “authenticity” in communication can be major drivers of success, but also noting how authenticity alone is not enough to explain the rather odd success of Steak-umm’s strategy, asking, rather pointedly, “If the brand had been communicating authentically for several years, why did we see such a spike in followers now?” Based on expectancy violation theory, the authors ask: “What themes are reflected in consumer tweets about Steak-umm and to what extent do reactions reflect positive expectancy violations?”
To answer this question, the authors conducted a thematic analysis of Tweets replying or mentioning Steak-umm from April 1 to 14, 2020, limiting them to US Twitter users posting in English. (It’s at this point that I must mention that I strongly suspect some of my own Tweets from back then are likely to have been included in this analysis, because I did respond to or quote-Tweet Steak-umm at least a few times during that period.) Doing that analysis according to Smith’s five step process for developing and collapsing themes, the most prominent themes found included praise, leadership, surprise, and product mentions.
I won’t rehash the praise that much, other than to mention that, unsurprisingly, there were indeed numerous Tweets praising Steak-umm for its Tweets about how to evaluate COVID-19 information and misinformation, how to evaluate self-proclaimed “experts” on social media writing about the pandemic, COVID-19 epidemiology, and potential treatments. Mine were among them, and many of them went something like this:
When you get more responsible information from a frozen meat than from the President of the United States.
Thank you, @steak_umm. For the yummy steak sandwiches and the info. https://t.co/gq2lAYuC2Z
— Amy Mode Activated ᓚᘏᗢ ಇ. (@ActivateAmyMode) April 10, 2020
And this:
Steak-umm offering the sensible, rational defense of truth, repudiation of opportunism, and call to humanity that we all need right now. https://t.co/UGAP5GQVsO
— Dr. Angela Rasmussen (@angie_rasmussen) April 13, 2020
Next up was leadership:
Another related theme was leadership. This theme was seen in several posts comparing its communication to that of President Trump. For example, one post said, “When you get more responsible information from a frozen meat than from the President of the United States …” (Toots the Red, 2020). It was also seen in posts saying Steak-umm, or the person behind the posts, is a hero, should run for president, or be nominated as vice president by presidential candidate Joe Biden. Examples included, “Umm, thank you, @steak_umm, for your sound reasoning. Exceptional read and quite right. Wanna run for President?” (Rau, 2020), and “Can we drop Biden and choose the @steak_umm intern instead?” (Wilson, 2020).
You get the idea. Amusingly, there was this response:
Indeed. It’s incredibly unlikely, of course, but a nice fantasy to have.
Obviously, to me, the big theme was surprise—pleasant surprise, but surprise nonetheless:
Surprise, confusion, and observations about the strangeness of 2020 were also prevalent. This came in the form of commenting on how it is odd that a frozen meat company is taking on this role and how that is a reflection of the strange reality of 2020 during the COVID-19 pandemic. Exemplar posts included, “Can’t believe I’m retweeting @steak_umm but these are strange times we’re living in” (Woods, 2020), and “@steak_umm being the voice of reason is peak 2020” (Jamie Plus a 6 Foot Radius of Empty Space, 2020).
You know, I had always wondered how one cites a Tweet in an academic paper. Now I know. In any case, these are indeed strange times we’re living in, and that’s why I subtitled this post “The Strange Case of Steak-umm”.
Consistent with Steak-umm’s Twitter feed’s main purpose, which is to sell frozen meat, there were also posts that explicitly helped to sell Steak-umm product:
Most immediately related to company success were posts that endorsed Steak-umm products or explicitly stated purchase intentions. For instance, “The only way I can thank Steak-umm for the truth bombs is to buy some Steak-umm, so I’ll make some cheesesteaks soon, yo” (Valholla, 2020). Some of these tweets reflected nostalgia for the brand and referred to childhood or college. One such tweet said, “I haven’t had @steak_umm since I was a wee lass but I’m definitely getting some next grocery trip because I’m in love with their social media presence right now” (Liliana, 2020).
Again, this just goes to show that it is possible to do well by doing good, and this sort of brand message amplification was, no doubt, part of what Steak-umm’s social media team had hoped to accomplish when it started its campaign.
On the other hand, although Bogomoletc and Lee didn’t mention it, I can’t help but note that not all the responses to its Tweets have been positive. For example:
Haters aside, Bogomoletc and Lee come up with some takeaways from this campaign:
This case study provides several takeaways for strategic and technical communicators. First, the study demonstrates that companies should not be afraid to surprise their publics and show a human side. Showing humanity might be especially relevant for communicating complex matters such as misinformation or other science communication topics (Steiner, 1999). In this case, that involved a professional communicator not only communicating complex information but also providing followers insight into the technical communication process in a way for which communication practitioners and scholars are uniquely situated.
Second, previous authentic communication demonstrating the commitment of a company to its values might be a foundation for ensuring positive outcomes of expectancy violation. This takeaway demonstrates that “values [are] important in our organizations of classrooms, workplaces, and situations beyond these settings in which texts are used to establish or change policies and priorities” (Rude, 2009, p. 182). Shared values mediated through Steak-umm’s texts were clearly more important for the brand’s stakeholders than was the established decorum that dictates certain behavioral norms for organizations in a crisis situation.
It really is saying something that one of the most effective voices on Twitter against COVID-19 misinformation is the Twitter account of a frozen meat company. Surprise, indeed. As much as I might have predicted many of the things that have happened during the pandemic that I discussed at the beginning of this post, I never would have predicted that a frozen meat company would come to occupy such a prominent position in the war against COVID-19 misinformation.
Advertisers took note, too, as this AdAge article from April demonstrates. It turns out that the voice behind Steak-umm’s social media is a man named Nathan Allebach, who describes himself in his Twitter bio as “covering internet culture wars, advertising, and conspiracy theories”. Allebach was brought into the company for this express purpose:
The man behind the curtain is Nathan Allebach, a social media manager at Allebach Communications. He and his team were brought into the Steak-umm family in 2015 with the express mission of reaching the media-savvy millennial audience and a single directive: innovate.
That innovation — which had already gained recognition in 2018 after pensive tweets waxing about millennial despondency — came to fruition in early April.
In other words, this isn’t the first time that the Steak-umm Twitter feed has achieved notoriety for a rather Zen-like (or, some would say, stoner-like) bit of Twitter artistry:
why are so many young people flocking to brands on social media for love, guidance, and attention? I'll tell you why. they're isolated from real communities, working service jobs they hate while barely making ends meat, and are living w/ unchecked personal/mental health problems
— Steak-umm (@steak_umm) September 26, 2018
they're crushed by student loan debt, disenfranchised by past generations, and are dreading the future of our world every day from mass media addiction and the struggle to not just be happy, but to survive this chaotic time with every problem happening at once under a microscope
— Steak-umm (@steak_umm) September 26, 2018
they grew up through the dawn of internet culture and have had mass advertising drilled into their media consumption, now they're being resold their childhoods by remakes, sequels, spinoffs, and other cheap nostalgia, making them more cynical to growth or authenticity
— Steak-umm (@steak_umm) September 26, 2018
they often don't have parents to talk to because they say stuff like "you don't know how good you have it," and they don't have mentors to talk to because most of them have no concept for growing up in this strange time, which perpetuates the feeling of helplessness/loneliness
— Steak-umm (@steak_umm) September 26, 2018
they have full access to social media and the information highway, but they feel more alone and insecure than ever. being behind a screen 24/7 has made them numb to everything, anxious and depressed about everything, and vitriolic or closed off toward anyone different from them
— Steak-umm (@steak_umm) September 26, 2018
young people today have it the best and the worst. there's so much to process and very few trusted, accessible outlets to process it all through. so they go to memes. they go to obscure or absurdist humor. they go to frozen meat companies on twitter. end rant
Steak-umm bless
— Steak-umm (@steak_umm) September 26, 2018
no soundcloud to add here. at the end of the day it’s easy to tweet about problems and complain about “the other,” it’s a lot harder to improve the self and work toward solutions
be encouraged and have hope my beeflings, the world needs it
— Steak-umm (@steak_umm) September 26, 2018
You can see from this thread from two years ago many of the same elements in Steak-umm’s Twitter threads about COVID-19, critical thinking, and science in 2020. There’s the self-deprecating, almost absurdist nature of the Tweets. Before the Tweets about Millennials, Steak-umm disrupted Twitter with a campaign to be “verified” by Twitter, the process that leads to those little checkmarks by the names of people on Twitter accounts, considered a badge of legitimacy that the social media company grants to company brands, journalists and famous people in the form of a blue check mark on their profile.
Basically, the 2020 Steak-umm Twitter campaign promoting critical thinking, particularly about COVID-19, is the longest and most sustained effort I’ve seen from the company. It’s gone on since early April and, as I showed above, is still ongoing. That being said, I never forget that this is an advertising campaign. It is undoubtedly an unconventional and innovative ad campaign that seeks to tie a brand to a societally useful and beneficial thing: Promoting skepticism and critical thinking about not just COVID-19, but everything.
Still, even while doing that, Steak-umm tells us that that’s what it’s doing. Indeed, it’s always been telling us that that’s what it’s doing. For example, later in April, Steak-umm opined:
note: all companies have a bottom line, so anything we publish is a form of propaganda to encourage positive association and memory with our brand, despite whatever our intentions. remember to consume advertising and PR with skepticism, even if the message is "helpful"
— Steak-umm (@steak_umm) April 17, 2020
That’s so meta, and I couldn’t have said it better myself, even at the risk of some out there labeling me a shill for big frozen meat. I would even say that you should consume what we lay down here at SBM with skepticism as well. We do our damnedest to stick to the science, but all of us have our biases and we’re human. We’re not perfect. We are, also, not trying to sell you anything other than SBM, and we’re not charging anything for what we’re selling other than a bit of your time.

Trust Me-I’m a Doctor
Steven Novella on April 30, 2025
Link
Trust is a fragile yet critical resource for any institution. At the end of the day, civilization is mostly built on a handshake and a mutual agreement to follow the rules. This includes trust that designated experts have the expertise they claim, are competent, and are acting appropriately in the interest of others, rather than exploiting their position for self-dealing. A critical loss of trust leads to the collapse of institutions and even nations.
This is why perhaps the most insidious and long lasting damage that will be done by the recent shakeup of our federal institutions, specifically with regard to healthcare, will be a loss of trust. A recent survey indicates this is already happening. At present, trust in government health institutions is still moderately high:
“Majorities of U.S. adults say they feel mostly positive toward the CDC (65%), their state public health department (60%), and their local public health department (63%). Majorities of adults across political parties also trust these agencies’ health recommendations a great deal or somewhat, including 77% who trust CDC recommendations, 80% who trust their state health department’s recommendations, and 82% who trust their local health department’s recommendations.”
I would argue that ideally these numbers should hover around 90% or higher, but I guess the bar has been set so low recently that a roughly two-thirds level of trust is considered good news. The survey also shows that the situation is likely to get worse:
“More than four in ten U.S. adults (44%) say having the new leaders in charge of federal public health agencies will make them trust health recommendations coming from these organizations less than they used to. Smaller shares say leadership changes will make them trust recommendations more (28%) or about the same (27%) as they used to. Views are divided along partisan lines, with most Democrats saying they will lose trust (76%) and a majority of Republicans saying they will gain trust (57%).”
It may seem obvious that there is a strong partisan reaction to new federal leadership – but that should not be the case when it comes to science, healthcare, or any area that is supposed to be based on objective data and not politics. When it comes to priorities, most Americans in this survey agree, at about the 80% level – we need to tackle chronic illness, improve infant mortality, address addiction, and improve water safety. We want the same things – the difference is in who we trust.
Making science partisan is one of the primary problems here (an issue that goes beyond healthcare). When experts become political appointees, how can they properly command trust. Trust has to be earned and maintained, and one of the ways to do that is to follow a merit-based and transparent process, not an arbitrary political one. Institutions like the CDC are supposed to be insulated from partisan politics.
We have spent a great deal of space recently writing about RFK Jr. as the new HHS secretary, because he is uniquely incompetent and dangerous in this position. But he is not a one-off or an aberration. Part of the reason his appointment deserves so much attention is because he represents the ultimate expression of trends we have been warning about at SBM since our creation.
One of those trends is the rise of conspiracy theories, which have always been simmering in the background of American culture but have been given rocket fuel by social media and algorithms designed to suck people down conspiracy rabbit holes. Conspiracy theories are all about not trusting authorities, experts, and those in power. It is a pathological distrust, a narrative that elevates distrust to the ultimate virtue.
Another trend has been the rise of so-called alternative medicine as if it were a valid endeavor (something other than just a rebranding of snake oil and health fraud). CAM also survives by sowing distrust in science, scientists, expertise, and institutions. It is an overt attempt to shift trust from legitimate expertise to gurus and the fringe.
CAM and conspiracy theories also work hand-in-hand with the healthcare freedom movement, which is ultimately about diverting trust away from regulations. This movement exploits the language of freedom, but really it is only about the freedom of con artists and snake oil peddlers to operate unimpeded by any consumer protection.
All of these trends are operating together and playing off of each other. Don’t trust experts, scientists, or healthcare professionals. Don’t trust the government or corporations. Don’t trust the media or the mainstream narrative. Trust only the gurus with the compelling narrative, telling you exactly what you want to hear and selling you false hope. RFK Jr. is the embodiment of all that.
What we are seeing is what happens when the conspiracy theorists and snake-oil peddlers are not just freed from any regulation, but are handed the reigns of regulation. There are many layers to the harm that will be done, but perhaps the greatest harm will be a death blow to any public trust in institutions.
The counternarrative will be that these institutions lost the public trust all on their own, but this is transparent nonsense. Sure, science and healthcare are hard, requiring complex calculations with imperfect data, and there will always be mistakes and malfeasance to point to. Many people point to the botching of our handling of COVID as an example, and I think it is a good example – but not in the way they intend.
The scientific and medical response to COVID, a novel virus causing a rapid pandemic, was actually quite impressive. All along the way experts admitted – this is a novel virus, there is still much we don’t know, we are building this plane as we are flying it, etc. They made the best judgements they could with the information they had erring on the side of saving lives.
And all along the way, sniping from the sidelines, were the conspiracy theorists, healthcare freedom advocates, gurus, and charlatans sowing as much distrust in the system as possible, turning everything into a conspiracy, hyping fake treatments, and stoking as much anger as possible. They demonized Anthony Fauci to put face on their conspiracy theories. Then they have the audacity to claim that the resulting lack of trust was the fault of the experts themselves. They used the distrust they cultivated to justify themselves.
I would never argue that science is perfect, that the institutions of medicine are beyond criticism, or that there weren’t mistakes in hindsight in how COVID was handled. Of course there were mistakes. This is hard, and we are imperfect.
This gets to one of the greatest challenges of being and advocate for SBM and scientific skepticism in general. Trust is complicated, and we do not advocate for blind trust or unlimited trust. But we need a mature, eyes-open, transparent trust (trust but verify). We need to recognize the value of expertise, of evidence-based regulations, and of standards. We need to trust in a logical and tested process. But we also need to maintain a healthy skepticism towards all of it, constantly question and examine the system, and explore ways to make it work better. This requires judgement, nuance, and hard work. It’s easier to just burn it all down and go with what feels good – and that’s what we are experiencing right now.

   Causation and Hill’s Criteria
Mark Crislip
Link
Causation is not so simple to determine as one would think. A mantra at SBM is ‘association is not causation’ and much of the belief in the efficacy of a variety of quack nostrums occurs because improvement occurs after use of a nostrum, therefore improvement occurs because of use of a nostrum. It is why vaccines as a cause of autism are so compelling to some. Vaccines are given at the same time autism starts to manifest. It would require more intellectual power than I have not to conclude, wrongly, that vaccines caused the autism. Concluding causation from sequential events is how the human mind works, and reality, as we know and ignore, constantly conspires to fool us into making false causal connections. In Infectious Diseases I see the error almost daily. The patient had a fever, patient was given antibiotics, fever went away. Therefore the antibiotics treated an infection. Well, maybe, maybe not. One of my mantras is ‘antibiotics are not antipyretics’ and you must be very careful before concluding that the fever went away because of the penacephalone.
I blog and podcast in large part to educate myself. It is amazing how much I learn in the process of preparing for one of the entries in my multimedia medical empire. Areas of knowledge that I had no idea existed can be revealed for exploration. I do not have formal training in epidemiology, although part of my job is hospital epidemiology. Like much of medicine, I acquired what I do know from on-the-job training and lots of reading. So imagine my surprise and delight to discover Hill’s Criteria of Causation, thanks to its application to chiropractic subluxation.
In 1965, Austin Bradford Hill published “The Environment and Disease: Association or Causation?” Dr. Hill, an occupational physician, sought to answer the questions (another discussion of the paper here):
How in the first place do we detect these relationships between sickness, injury and conditions of work? How do we determine what are physical, chemical and psychological hazards of occupation, and in particular those that are rare and not easily recognized?…In other words we see that the event B is associated with the environmental feature A, that, to take a specific example, some form of respiratory illness is associated with a dust in the environment. In what circumstances can we pass from this observed association to a verdict of causation? Upon what basis should be proceed to do so?
He then proceeds to list criteria (he refers to them as viewpoints) that help in determining causation. I like frameworks for thinking about processes. They provide a starting point for considering problems. Becoming a doctor is, in part, internalizing frameworks. Early in training you carry lists and papers that remind you how to evaluate acidosis, or the physiology of heart failure or suspected meningitis. After time and repetition these lists are internalized and you can evaluate the problems without resorting to lists. As an intern one of the papers I carried around was The New England Journal of Medicine article on the physiology of the Swan-Ganz catheter, and I would refer to it with each patient who had a Swan. One day I did not need to refer to the paper. I had internalized the information, and tossed the paper into the trash. Frameworks are not the be all and end all, but do serve as foundation upon which to build ideas. Part of being a specialist is to recognize when the framework doesn’t apply. However, I never had a formal framework for thinking about what constitutes causality in medicine. It is worth reading in the original if no other reason as an appreciation of a time when the medical literature was not dry as dust and devoid of humor and style. Current medical journal writing is often an excellent replacement for Ambien, even when you are fascinated by the topic. To call it the medical ‘literature’ is to refer to the phone book as literature.
The viewpoints to consider in determining if association is due to causation:
1) Strength. How strong is the association between the cause and the effect? Hill uses the example of chimney sweeps, who died of scrotal cancer at rates 400 times the normal population. It killed Bert, or so I was lead to believe. He points out that a strong association like scrotal cancer and chimney sweeps is good evidence in favor of causality from an environmental exposure. He also points to that small effects in a population can still be considered strong associations and uses the death rates from Snows evaluation of the 1855 cholera outbreak, where the death rates from the contaminated water were 17/10,000 vs. 5/10,000 in the general population. Not a huge increase in mortality, but a strong association none the less.
The strength of ‘alternative’ therapies usually hovers around background noise, usually at the level of personal experience. If acupuncture or homeopathy were 400 times superior to placebo, there would no discussion of its validity. Many medical therapies are not 400 times as effective as placebo, but the strength of the association between cause and effect is well above background noise.
2) Consistency. Almost every study should support the association for there to be causation. He uses the example of cigarettes as a cause of lung cancer, “The Advisory Committee to the Surgeon-General of the United States Public Health Service found the association of smoking with cancer of the lung in 29 retrospective and 7 prospective inquiries.” From the vantage point of 2009, where the carcinogenic effects of cigarettes are well established, this example is amusing, but instructive. Over time, as studies progressed, there was a consistent association between smoking and cancer.
He also warns about the importance of a good control group, the inclusion of which often spells the death of ‘alternative’ therapy efficacy, as the recent studies in acupuncture have demonstrated:
Patients admitted to hospital for operation for peptic ulcer are questioned about recent domestic anxieties or crises that may have precipitated the acute illness. As controls, patients admitted for operation for a simple hernia are similarly quizzed. But, as Heady points out, the two groups may not be in pari materia. If your wife ran off with the lodger last week you still have to take your perforated ulcer to hospital without delay. But with a hernia you might prefer to stay at home for a while‚ to mourn (or celebrate) the event. No number of exact repetitions would remove or necessarily reveal that fallacy.
As readers of this blog are probably aware, there is one consistent result in ‘alternative’ therapies: increasing the quality of the study decreases the efficacy until the best studies show no effect.
3) Specificity. Since diseases can have multiple etiologies and therapies can have multiple effects, this is a weaker criteria. However, given the knowledge of physiology and biochemistry since 1965, we have more sophisticated techniques for measuring and determining specificity. From the perspective of opportunistic infections, with no knowledge of viral pathophysiology, HIV is hardly a specific cause of disease. GIven the ability to measure HIV viral load and an understanding of the consequences of HIV depletion of CD4 cells, HIV has high specificity for causing AIDS.
“In short, if specificity exists we may be able to draw conclusions without hesitation; if it is not apparent, we are not thereby necessarily left sitting irresolutely on the fence.”
As Dr Hall has discussed, many ‘alternative’ medical paradigms completely lack specificity and are the one true cause or treatment of all diseases, be it subluxation, a liver fluke, or colonic toxin build up. Fools all; infections are the one true cause of all disease.
4) Temporality. The order should be exposure, disease, treatment, resolution. Cause should proceed effect.
Does a particular diet lead to disease or do the early stages of the disease lead to those particular dietetic habits? Does a particular occupation or occupational environment promote infection by the tubercle bacillus or are the men and women who select that kind of work more liable to contract tuberculosis whatever the environment‚ or, indeed, have they already contracted it? This temporal problem may not arise often, but it certainly needs to be remembered
5) Biological gradient. Also known as dose-response. A little exposure should result in a little effect, a large exposure should cause a large effect. Certainly well known to anyone who drinks alcohol; I suppose all homeopaths must be teetotallers.
The comparison would be weakened, though not necessarily destroyed, if it depended upon, say, a much heavier death rate in light smokers and a lower rate in heavier smokers. We should then need to envisage some much more complex relationship to satisfy the cause and effect hypothesis. The clear dose-response curve admits of a simple explanation and obviously puts the case in a clearer light.
Most ‘alternative’ therapies are binary and have no gradient of effect. I suppose that a chiropractor could say your spine is partly unsubluxed as a result of half a spinal manipulation or an acupuncturist saying, your chi is partly unblocked as I used too few needles. I suppose. I assume a reader will comment on the validity of this observation.
6) Plausibility. The effect must have biologic plausibility. I would take it a slightly differently: not only should it be biologically plausible, but should not violate well known laws of the universe. Hill points out “but this is a feature I am convinced we cannot demand. What is biologically plausible depends upon the biological knowledge of the day.”
Yet there is a difference between what is not yet known but possible — for example Helicobacter as a cause of gastric ulcers, which is odd but not impossible — and what almost certainly will never known because it cannot exist without a radical rewrite of all of science: meridians or water memory, which are odd and impossible. I know that there are more things in heaven and earth than are dreamt of in my philosophy. But you have to prove it to me.
In short, the association we observe may be one new to science or medicine and we must not dismiss it too light-heartedly as just too odd. As Sherlock Holmes advised Dr. Watson, “when you have eliminated the impossible, whatever remains, however improbable, must be the truth.” A nice quote, but not necessarily the case. Sometimes what remains is, however improbable, still nonsense. Which leads to:
7) Coherence. “On the other hand the cause-and-effect interpretation of our data should not seriously conflict with the generally known facts of the natural history and biology of the disease.”
I have discussed in prior entries that those who call themselves ‘holistic’ rarely are and a good physician understands a disease from the microscopic to the entire world. I know cholera, for example, from the level of the effect of the toxin on cellular receptors to the world wide changes in potable water that lead to the spread of disease and much in between. There is a coherence of understanding of the disease.
In a wider field John Snow‚ epidemiological observations on the conveyance of cholera by water from the Broad Street Pump would have been put almost beyond dispute if Robert Koch had been then around to isolate the vibrio from the baby nappies, the well itself and the gentleman in delicate health from Brighton. Yet the fact that Koch‚ work was to be awaited another thirty years did not really weaken the epidemiological case though it made it more difficult to establish against the criticisms of the day.
Ignoring most anatomy or physiology or other biologic understanding, most ‘alternative’ therapies have no coherence when placed in the context of the known universe. Homeopathy is, above all, totally incoherent.
8) Experiment. Always nice. Written in 1965, before the massive increase in biomedical research funding, experiments were not as vital in understanding diseases and treatments as they are today.
Unfortunately for most ‘alternative’ medicine, experiments rarely support their theory or efficacy. Not that it ever matters to the practitioners. I think about how my practice has changed over the last 25 years: adding, subtracting and modifying what I do as the data comes in. Consider the 44,000 articles in Pubmed in infectious disease that are published last year. I wonder how much chiropractic (233 articles on 2009) or acupuncture (1000 articles in 2009) or naturopathy (19 articles in 2009) or homeopathy (162 articles in 2009) practice changed as a result of published studies. It cannot be all that hard to keep up and, so, change accordingly.
9) Analogy. If one virus, for example, can cause a disease, then it is reasonable to suggest that a second virus could be responsible for a similar disease. Analogy is not the same as metaphor: both are the preferred methods of understanding ‘alternative’ therapies, but with little comparison to objective reality. He clearly states these are guidelines, and not to be followed blindly.
What I do not believe‚ and this has been suggested, that we can usefully lay down some hard-and-fast rules of evidence that must be obeyed before we can accept cause and effect. None of my nine viewpoints can bring indisputable evidence for or against the cause-and-effect hypothesis and none can be required as a sine qua non. What they can do, with greater or less strength, is to help us to make up our minds on the fundamental question, is there any other way of explaining the set of facts before us, is there any other answer equally, or more, likely than cause and effect?
The importance of considering all the data, the preponderance of information, in deciding cause and effect.
Hill is also not enthusiastic about statistics, the dreaded p-value:
No formal tests of significance can answer those questions. Such tests can, and should, remind us of the effects that the play of chance can create, and they will instruct us in the likely magnitude of those effects. Beyond that they contribute nothing to the, proof‚ of our hypothesis … Between the two world wars there was a strong case for emphasizing to the clinician and other research workers the importance of not overlooking the effects of the play of chance upon their data. Perhaps too often generalities were based upon two men and a laboratory dog while the treatment of choice was deducted from a difference between two bedfuls of patients and might easily have no true meaning. It was therefore a useful corrective for statisticians to stress, and to teach the needs for, tests of significance merely to serve as guides to caution before drawing a conclusion, before inflating the particular to the general… Yet there are innumerable situations in which (tests of significance) are totally unnecessary‚ because the difference is grotesquely obvious, because it is negligible, or because, whether it be formally significant or not, it is too small to be of any practical importance… What is worse the glitter of the t table diverts attention from the inadequacies of the fare.
Statistically significant nonsense is still nonsense. The article puts into perspective the ongoing problem of the meta-analysis. I always say that the meta-analysis is good for a general understanding of an intervention but rarely provides definitive answers. When meta-analyses are compared to subsequent randomized controlled trials, the meta-analyses got it wrong 35% of the time. As a result, I think meta-analyses are great if they support your prior beliefs and can be safely ignored if they contradict them. That is the problem with meta-analyses, they are good at the mathematics/statistics of multiple studies but fail to take into consideration the other viewpoints as enumerated by Dr. Hill. Far be it from me to suggest that the Cochrane reviews may be wanting as they are often considered the be all end all of analysis, but their reviews in the few areas I know a little about always leave me unsatisfied.
Dr Hill ends with a discussion on the importance of then using the information of when association merges into causation and to consider at what point we need to act on the information. Stopping a nausea medication because it may cause birth defects has a different impact than stopping the burning of fuels in the home as cause of lung disease. In the end we have to act on our information, even if it is always incomplete.
All scientific work is incomplete, whether it be observational or experimental. All scientific work is liable to be upset or modified by advancing knowledge. That does not confer upon us a freedom to ignore the knowledge we already have, or to postpone the action that it appears to demand at a given time.
Who knows, asked Robert Browning, but the world may end tonight? True, but on available evidence most of us make ready to commute on 8:30 the next day.
8:30. It was a more leisurely era.
The explicit application of Hill’s criteria is uncommon, at least if ‘Hill’s criteria’ is used as search criteria on Pubmed. There are areas on medicine that are not clearcut as to causality. Do some therapies work? And if so, in what populations? Is X the cause of disease Y?
Recently, Hill’s Criteria were applied to chiropractic subluxation, and subluxation was found wanting. In Chiropractic theory, spinal subluxation is considered to be the cause of nearly all disease. As readers of this blog know, subluxation was made up by DD Palmer and
The Association of Chiropractic Colleges paradigm statement (ACC Paradigm) suggested that, chiropractic is concerned with the preservation and restoration of health, and focuses particular attention on the subluxation. It also defined a subluxation as “a complex of functional and/or structural and/or pathological articular changes that compromise neural integrity and may influence organ system function and general health.
Subluxation is the basis of the practice of chiropractic. It there is no subluxation, the raison d’etre for chiropractic disappears with a pop or perhaps a crack.
So when Timothy A Mirtz et. al searched the literature looking for support for the most fundamental support for the practice of chiropractic and came up with bupkis, well, imagine if the literature search for germs as a cause of infection or atherosclerosis as a cause of heart attack and found nothing. Zip. Nil. Nada. Diddley squat.
What would happen to Infectious Diseases or Cardiology? I would hope they would disappear as specialty. If there is no basis for the practice, it should be abandoned. Right?
What is the support for subluxation, using Hills ever so helpful viewpoints? I will summarize with the table from article
As the conclusion states,
There is a significant lack of evidence in the literature to fulfill Hill’s criteria of causation as regards chiropractic subluxation. No supportive evidence is found for the chiropractic subluxation being associated with any disease process or of creating suboptimal health conditions requiring intervention. Regardless of popular appeal this leaves the subluxation construct in the realm of unsupported speculation. This lack of supportive evidence suggests the subluxation construct has no valid clinical applicability.
So why continue with chiropractic, at least based on the treatment of subluxation? Got me. And pay for it with tax dollars or insurance premiums? If chiropractic is based entirely on nothing substantial, then nothing is what should reasonably be paid.
One wonders about other alternative therapies: homeopathy, acupuncture, various energy therapies etc. Even if a meta-analysis demonstrated marginal statistical benefit, when Hills viewpoints are considered, I doubt any would hold up.

The misuse and abuse of meta-analyses
The meta-analysis has been misused and weaponized to promote dubious COVID-19 treatments.
Scott Gavura on November 25, 2021
Link


Today’s post is a short one. The idea emerged as I read through the tireless blogging from David Gorski and others and the work of epidemiologists like Gideon Meyerowitz-Katz who have painstakingly gone through the ivermectin literature, trial-by-trial, even scrutinizing patient-level data, to uncover some truly poor science (some of which appears fraudulent). Through their work, they’ve exposed how research studies are being misrepresented, even weaponized, to promote the use of ivermectin.
This post is not about ivermectin. It is about the tools we use to bring some sense to the medical literature – and ways in which casual readers, and even seasoned professionals, can be misled. I want to summarize some of the methods used to compile and summarize evidence, and the approaches that can be used reduce biases and improve the quality of our conclusions and decisions about our health. 
Finding good evidence
To guide decision-making in medicine, we need effective methods to identify, compile, and analyze volumes of information. The amount of evidence that’s generated on a daily basis is near-impossible for anyone to keep up with, even if the specific topic you’re interested in is niche or narrow. New evidence can be unclear and even contradictory, and the “best evidence” can change quickly. Just look at how the pandemic guidance has changed since early 2020, from the utility of masks, the relevance of handwashing, and how to manage (what is now recognized as) an airborne infection.
Let’s say you’re interested in therapies to treat COVID-19 infections. The World Health Organization has cataloged over 334,000 publications in its COVID-19 database. If I restrict my search just to clinical trials for COVID-19 treatments, I’m still left with over 1,000 papers, with more emerging daily. How do we identify the high quality, most relevant evidence from weaker evidence – and how should we try to wrap our head around the totality of all of this data?
A single clinical trial is usually insufficient (though not impossible) to change clinical practice. That’s often because there are so many considerations when asking a specific clinical question and designing a trial. What dose? What patient population? What is the treatment protocol? What is the endpoint that’s being measured? How large was the trial? And was the trial conducted in a way to be a truly “fair test” of a treatment – meaning blinding and appropriate controls? Several trials, may be conducted by different investigators, with a mix of findings. What’s needed is a method to identify the best of the best, and to try to synthesize what may appear to be contradictory findings.
In a lot of cases, we rely on the work of others to give us that perspective. Narrative reviews (like a blog post!) are the quickest and easiest, but the most prone to bias, because the author can cherry-pick the evidence to comment on:

A narrative review tends to be a discussion of a topic, where the author decides what evidence is relevant and worth discussing. They may be written by experts who have a deep understanding of the topic, and will summarize pertinent data and make general conclusions. This is not to say that narrative reviews are necessarily inaccurate – only that they can be prone to biases that may not be easy to spot. A narrative review leaves the reader totally in the hands of the author. For that reason, reading one or more narrative reviews can be reasonable starting points to understanding a topic, but cannot be considered comprehensive or definitive summaries of the evidence base of a topic or clinical question.
Systematic reviews and meta-analyses
Sometimes we want to know what the best evidence says. The opposite of the narrative review is the systematic review. That’s where the evidence is formally and systematically searched and analyzed.
A systematic review is a summary of the evidence that addresses a specific question and uses explicit, rigorous, and transparent methods to identify, select, appraise, and combine research studies. The included trials must be identified from a systematic search, with pre-defined quality criteria, that should be published in advance of the search itself. Researchers will identify the search strategy, the selection criteria (which trials will be included) and the analytic methods that will be applied. Doing all this in advance reduces the risk of bias when the data is actually analyzed. PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) defines the minimum data requirements that should be reported in a systematic review, improving the quality, consistency, and transparency of reviews.
A meta-analysis can be part of a systematic review, but can also be done on its own. (A systematic review appraises data before it is combined and analyzed in a meta-analysis.) It is a statistical compilation of multiple studies that examine the same topic, and it can be a useful tool as part of a systematic review. Done carefully, then can give a “point estimate” (aka “best guess”) of an effect that may be more precise than any of the individual trials that were included. But just as individual clinical trials can be biased, so can meta-analyses. Meta-analyses cannot correct for fundamentally poor source data. That is, they cannot glean accurate information from flawed research. The “garbage in, garbage out” adage applies here. Moreover, if different trials measured different endpoints, then they cannot be combined in a meta-analysis. Or you’ll see something kind of like this:

This is statistical heterogeneity, as discussed in a post by Dr. William Paolo back in August:
When analyzing the results of a meta-analysis it is imperative to focus on the degree of statistical heterogeneity to determine if the resultant point estimate of the compiled studies are, in fact, similar enough to be combined into a singular parsimonious estimate of effect. Theoretically, multiple smaller studies can be pooled and analyzed as a singular body of evidence if the component studies are of analogous design and methodology (allocation concealment, randomization, intention to treat analysis etc.) or rejected if this is not the case. It is required that the included studies be the same in terms of design, inclusion and exclusion criteria, demographics, equivalent doses of an intervention, outcomes, and a myriad of other factors to comfortably combine said studies into a unitary body of evidence. Concordant with this is the expectation that if the included studies of a meta-analysis are of equal kind that experimentally derived results, taken under similar conditions, should roughly approximate one another. In other words, all things being equal, the same exact experiment done in multiple places by different people should roughly find the same results if the studies are in fact similar enough to be analyzed as one.
Ivermectin trial alchemy
With the above in mind, let’s take a closer look at how the meta-analysis has been misused in the pandemic. Ivmmeta.com is a site that is frequently cited by advocates of ivermectin which, as David Gorski first discussed way back in June, seemed to brought to us by the people that created hcqmeta.com (Remember hydroxychloroquine? How vintage.) It’s impressive, in an information-overload kind of way, claiming to be a “real-time meta-analysis of 67 studies”. Rather than go through the criticisms of the website in detail, I will point you to the writing and blogging of Gideon Meyerowitz-Katz who has scrutinized the data presented – and the website’s response to his initial criticisms.
In something of a hilarious development, the ivmmeta authors have added a section titled "with GMK exclusions" to their website
Unfortunately, this still includes endless awful studies. Let's take a look at some of the papers that remain on this website https://t.co/moT5tNnD1n
— Health Nerd (@GidMK) October 26, 2021
In short, ivmmeta.com (or sister sites, like c19ivermectin.com) does not adequately take into account the quality and diversity of the data it is combining. As David Gorski highlighted, there are warnings in the actual literature about this:
Different websites (such as https://ivmmeta.com/, https://c19ivermectin.com/, https://tratamientotemprano.org/estudios-ivermectina/, among others) have conducted meta-analyses with ivermectin studies, showing unpublished colourful forest plots which rapidly gained public acknowledgement and were disseminated via social media, without following any methodological or report guidelines. These websites do not include protocol registration with methods, search strategies, inclusion criteria, quality assessment of the included studies nor the certainty of the evidence of the pooled estimates. Prospective registration of systematic reviews with or without meta-analysis protocols is a key feature for providing transparency in the review process and ensuring protection against reporting biases, by revealing differences between the methods or outcomes reported in the published review and those planned in the registered protocol. These websites show pooled estimates suggesting significant benefits with ivermectin, which has resulted in confusion for clinicians, patients and even decision-makers. This is usually a problem when performing meta-analyses which are not based in rigorous systematic reviews, often leading to spread spurious or fallacious findings.36
You might be asking at this point – has anyone actually done a robust systematic review of ivermectin? The answer is yes, and the results will not surprise (spoiler: ivermectin doesn’t seem to work).
A tool, not a quality indicator
        ◦ All types of data can be combined into a meta-analysis. When used to study high-quality trials of similar populations that used similar methods, they can be powerful tools. But a meta-analysis is just a tool, and cannot make gold out of lead. Combining poor quality studies that are not sufficiently similar in a meta-analysis does not give us any useful information. Moreover, in the absence of rigorous, transparent, and pre-specified rules, meta-analyses are prone to bias, just like the source clinical trials themselves. Meta-analyses prepared outside the framework of a rigorous systematic review should be viewed with caution.

Philosophy Meets Medicine
Harriet Hall on December 3, 2013
Link
Note: This was written as a book review for Skeptical Inquirer magazine and will be published in its Jan/Feb 2014 issue.
————–
Medicine is chock-full of philosophy and doesn’t know it. Mario Bunge, a philosopher, physicist, and CSI (Center for Skeptical Inquiry) fellow, wants to bring philosophy and medicine together for mutual benefit. He has written a book full of insight and wisdom, Medical Philosophy: Conceptual Issues in Medicine.
Whether doctors recognize it or not, medicine is firmly based on the philosophical principles of materialism, systemism, realism, scientism, and humanism. Bunge explains that:
Without materialism, both diseases and therapies would be taken to be purely spiritual.
Without systemism, every disease would be attributed to an independent module.
Without realism, diseases would be viewed as either imaginary or as social flaws.
Without scientism, either nihilism or dogmatism would prevail, and all the achievements of biomedical research of the last 500 years would be consigned to oblivion.
Without humanism, all medical practice would be mercenary, and there would be no public health care.
He points out that all these scenarios are alive and well elsewhere. Under the guise of “alternative medicine,” ancient prescientific medical systems are still being practiced nearly everywhere “to the detriment of people, the flora, and the fauna.” CAMs (complementary and alternative medicines) are isolated bodies of belief that are not based on science and do not examine their own philosophical presuppositions. Like religion, alternative medicine relies on the gullibility of its clients, who have been educated to believe without evidence and without explanation. Bunge says that acupuncture is useless except as an analgesic placebo, that the Ayurvedic pharmacopeia is fantastic, that only a few of the 11,000 Chinese herbal medicines have ever been subjected to controlled clinical trials, that fully 1/5 of the most commonly prescribed Japanese medicinal herbs are toxic, that homeopathy ignores the whole of science, and that naturopathy is only marginally better in that it combines an ignorance of the scientific method with remedies in amounts that might have actual effects on the body. The CAMs have contributed nothing true or useful to health care. He discourages any further research on CAM hypotheses that are incompatible with the bulk of scientific knowledge.
The concept of vitalism is now obsolete. Reductionists postulated some kind of supernatural vital spark because they couldn’t imagine how the whole could have properties not present in the parts; today we understand that new properties are emergent and develop by purely natural processes. A remnant of reductionism is seen in DNAism, the simplistic idea that everything is reducible to DNA. Understanding complex facts in biology requires both a top-down and a bottom-up approach. We exist on several levels, and conditions like obesity and addiction are affected by social factors, inadaptations, and wrong decisions. Social and environmental factors can affect the action of genes through epigenetics and alterations of gene expression.
In medicine, “it is systems all the way.” Holism sees a forest but not the trees. Individualism sees a single tree. Systemism recognizes that there is no whole without its parts and that some wholes have global properties that their parts lack.
Bunge throws another wrench into intelligent design: if our body were wise, it wouldn’t get sick except through accident. The body produces antibodies for protection against infections, but it also produces endotoxins and maintains cancer cells.
He covers the history of medicine and the emergence of science, defined by Richard Feynman as “the belief in the ignorance of experts.” He laments the way so many people confuse frequencies with probabilities: there isn’t a 30% probability that you have a disease, you either have it or you don’t.
He covers the scientific method, why randomized controlled trials (RCTs) are necessary but insufficient, and why most published research findings are false. He explains that the placebo effect is not evidence of the power of an immaterial mind over the material body, but just the opposite: the effect of a material cerebral cortex on material subcortical organs.
Like those of us on the Science-Based Medicine blog, he is a critic of evidence-based medicine (EBM). He complains that evidence-based medicine is not as novel as its champions claim it to be and that it exaggerates the weight of evidence at the expense of that of hypothesis. EBM “has only strengthened the empiricist tendency to accumulate undigested data and mistrust all theory.” Science only advances when experiments are guided by hypotheses. Those who hold that passing a randomized controlled trial is all we need for proof are stuck in empiricism. RCTs are the gold standard, but there is a platinum standard (RCTs plus mechanisms of action) that is better. And even studies that fit the platinum standard should not be idolized.
Preventive medicine is arguably the most important medicine of all, and it is a social issue. Secular humanism is the only ethical system that defines “justice” as the balance of rights with duties. Everyone has the moral right to medical assistance. This implies a moral obligation to providing sanitation, vaccination, and access to health care for those who can’t afford it. Today, modern pharmacology only reaches 2 out of every 7 human beings. Libertarians would let individuals make their own decisions about vaccination, but in health matters, individualism just doesn’t fly. Sanitary systems, compulsory vaccination, and epidemic management require action by society as a whole.
Bunge calls for a systemic approach to solve problems at the intersection between medicine and politics; he proposes a whole package of social policies with a multifaceted program to address access to health care, planned parenthood, more effective preventive measures including better education about diet and hygiene, income inequality, housing, environmental protection, price control of pharmaceutical drugs, reinforcing NGOs, and more participation of the citizenry in politics. Even if you don’t agree with his solutions, you will probably agree with his identification of the problems.
This is not a “fun” read or an easy one, but it is packed with information and with food for thought about serious issues that affect every one of us. It is a challenging book that will make readers re-consider their unconscious assumptions, confront new ideas about things they haven’t thought about, and think more deeply about things they may have taken for granted. Oliver Wendell Holmes, Sr. said, “A mind that is stretched by a new experience can never go back to its old dimensions.” This is a mind-stretching book.
———–
Followup note: This book gave me a new perspective on the term “holistic.” I had interpreted it as being similar to the conventional medical approach of looking at all aspects of a patient, treating the whole patient rather than just one body part; but according to Bunge it means seeing the forest only as a whole, without paying any attention to individual trees. What conventional medicine does is what he calls systemism, recognizing global aspects as well as individual parts. CAM “holism” can mean seeing only the global aspects and ignoring basic details like anatomy, physiology, and biochemistry. Good medicine has always encompassed both the details and the emergent global properties, seeing the whole forest along with the details of every tree. It is holistic while simultaneously being individualistic; in a sense, it is more “holistic” than so-called holistic CAM: there can be no whole without its parts.

   Certainty versus knowledge in medicine
David Gorski
Link
I don’t want knowledge. I want certainty!
— David Bowie, from Law (Earthlings on Fire)
If there’s a trait among humans that seems universal, it appears to be an unquenchable thirst for certainty. It is likely to be a major force that drives people into the arms of religion, even radical religions that have clearly irrational views, such as the idea that flying planes into large buildings and killing thousands of people is a one-way ticket to heaven. However, this craving for certainty isn’t expressed only by religiosity. As anyone who accepts science as the basis of medical therapy knows, there’s a lot of the same psychology going on in medicine as well. This should come as no surprise to those committed to science-based medicine because there is a profound conflict between our human desire for certainty and the uncertainty that is always inherent in so much of our medical knowledge. The reason is that the conclusions of science are always provisional, and those of science-based medicine arguably even more so than many other branches of science.
In fact, one of the hardest things for many people to accept about science-based medicine is that the conclusions of science are always subject to change based on new evidence, sometimes so much so that even those of us “in the biz” can become a bit disconcerted at the rate at which knowledge we had thought to be secure changes. For example, think of how duodenal peptic ulcer disease was treated 25 years ago and then think about how it is treated now. Between 1984 and 1994, a revolution occurred on the basis of the discovery of H. pylori as the cause of most of the gastric and peptic ulcer disease we see. Where in 1985 we treated PUD with H2-blockers and other drugs designed to block gastric acid secretion, now antibiotics represent the mainstay of treatment and are curative at a much higher success rate than any treatment other than surgery and without the complications of surgery. I’m sure any other physician here could come up with multiple other examples. In my own field of breast cancer surgery, I look back at how we treated breast cancer 22 years ago, when I first started residency, and how we treat it now, and I marvel at the changes. If such changes can be disconcerting even to physicians dedicated to science-based medicine, imagine how much more disconcerting they are to lay people, particularly when they hear news reports of one study that produces one result, followed just months later by a report of a different study that gives a completely different result.
We see this phenomenon of craving certainty writ large and in bold letters in huge swaths of so-called “alternative” medicine. Indeed, a lot of quackery, if not most of it, involves substituting the certainty of belief for the provisional nature of science in science-based medicine, as well as the uncertainty in our ability to predict treatment outcomes, particularly in serious diseases with variable biology, like several types of cancer.
Examples abound. Perhaps my favorite two examples include Hulda Clark, who attributed all cancer and serious disease to a common liver fluke, and Robert O. Young, who believes that virtually all disease is due to “excess acid.” So prevalent is this tendency that Harriet Hall once skewered it in a delightful post entitled The One True Cause of All Disease, where she listed a rather large samplings of things that various quacks have implicated as “the one true cause” of various diseases — or all diseases.
Time and time again, if you look carefully at “alt-med” concepts and the therapies that derive from those concepts, you find utter simplicity (or, more appropriately in many cases, simple-mindedness) tarted up with complicated-sounding jargon. Homeopathy, for instance, is at its heart nothing more than sympathetic magic, with its concept of “like cures like,” combined with the principle of contagion, with its concept that water somehow has a “memory” of the therapeutic substances with which it’s come in contact but, as Tim Minchin so hilariously put it, “it somehow forgets all the poo it’s had in it.” Reiki and other “energy healing” modalities can be summed up as “wishing makes it so,” with “intent” having the power to manipulate some fantastical life energy to heal people. It’s faith healing, pure and simple.
The simplicity of these concepts at their core makes them stubbornly resistant to evidence. Indeed, when scientific evidence meets a strong belief, the evidence usually loses. In some cases, it does more than just lose; the scientific evidence only hardens the position of believers. We see this very commonly in the anti-vaccine movement, where the more evidence is presented against a vaccine-autism link, seemingly the more deeply anti-vaccine activists dig their heels in to resist, cherry picking and twisting evidence, launching ad hominem attacks on their foes, and moving the goalposts faster than science can kick the evidence ball through the uprights. The same is true for any number of pseudoscientific beliefs. We see it all the time in quackery, where even failure of the tumor to shrink in response can lead patients to conclude that the tumor, although still there, still can’t hurt them. 9/11 Truthers, creationists, Holocaust deniers, moon hoaxers — they all engage in the same sort of desperate resistance to science.
Even those who in general accept science-based medicine can be prone to the same tendency to dismiss evidence that conflicts with their beliefs. A while back, I saw an article by Christie Aschwanden discussing just this problem. The article was entitled Convincing the Public to Accept New Medical Guidelines, and I feel it could almost have been written by Mark Crislip or myself, only minus Mark’s inimitable self-deprecating yet cutting sarcasm, or my own alleged talent for nastiness and ad hominem. (I guess I need to become more cuddly.) To set up its point that persuading people to accept the results of new medical science is exceedingly difficult, the article starts with the example of long distance runners who believe that taking ibuprofen (or “vitamin I”) before a long run reduces their pain and inflammation resulting from the run:
They call it “vitamin I.” Among runners of ultra-long-distance races, ibuprofen use is so common that when scientist David Nieman tried to study the drug’s use at the Western States Endurance Run in California’s Sierra Nevada mountains he could hardly find participants willing to run the grueling 100-mile race without it.
Nieman, director of the Human Performance Lab at Appalachian State University, eventually did recruit the subjects he needed for the study, comparing pain and inflammation in runners who took ibuprofen during the race with those who didn’t, and the results were unequivocal. Ibuprofen failed to reduce muscle pain or soreness, and blood tests revealed that ibuprofen takers actually experienced greater levels of inflammation than those who eschewed the drug. “There is absolutely no reason for runners to be using ibuprofen,” Nieman says.
The following year, Nieman returned to the Western States race and presented his findings to runners. Afterward, he asked whether his study results would change their habits. The answer was a resounding no. “They really, really think it’s helping,” Nieman says. “Even in the face of data showing that it doesn’t help, they still use it.”
As is pointed out, this is no anomaly. Aschwanden uses as another example a topic that’s become a favorite of mine over the last six months or so since the USPSTF released revised guidelines for mammographic screening. Take a look at what she says about the reaction:
This recommendation, along with the call for mammograms in women age 50 and older to be done every two years, rather than annually, seemed like a radical change to many observers. Oncologist Marisa C. Weiss, founder of Breastcancer.org, called the guidelines “a huge step backwards.” If the new guidelines are adopted, “Countless American women may die needlessly from breast cancer,” the American College of Radiology said.
“We got letters saying we have blood on our hands,” says Barbara Brenner, a breast cancer survivor and executive director of the San Francisco advocacy group Breast Cancer Action, which joined several other advocacy groups in backing the new recommendations. Brenner says the new guidelines strike a reasonable balance between mammography’s risks and benefits.
I discussed the guidelines in considerable detail twice. Let’s put it this way: I’m in the business, so to speak, and even I was shocked at the vehement reactions, not just from patients and patient advocacy groups, whose reaction I could completely understand (after many years of hearing that beginning mammography at age 40 was critical to save lives), but even from some of my very own colleagues. I was particularly disgusted by the reaction of the American College of Radiology, which was nothing more than blatant fear mongering that intentionally frightened women into thinking that the new guidelines would lead to their deaths from breast cancer. As much as we’d like to pretend otherwise, even science-based medical practitioners can fall prey to craving the certainty of known and accepted guidelines over the uncertainty of the new. And if it’s so hard to get physicians to accept new guidelines and new science, imagine how hard it is to get patients to accept them.
There is abundant evidence of how humans defend their views against evidence that would contradict them, and it’s not just observational evidence that you or I see every day. Scientists often fall prey to what University of California, Berkeley, social psychologist Robert J. MacCoun calls the “truth wins” assumption. This assumption, stated simply, is that when the truth is correctly stated it will be universally recognized. Those of us who make it one of our major activities to combat pseudoscience know, of course, that the truth doesn’t always win. Quite the contrary, actually, I’m not even sure the “truth” wins a majority of the time — or even close to a majority of the time. Moreover, most recommendations of science-based medicine are not “truth” per se; they are simply the best recommendations physicians can currently make based on current scientific evidence. Be that as it may, the problem with the “truth wins” viewpoint is that the “truth” often runs into a buzz saw known as a phenomenon that philosophers call naive realism. This phenomenon, boiled down to its essence is the belief that whatever one believes, one believes it simply because it’s true. In the service of naive realism, we all construct mental models that help us make sense of the world. When the “truth wins” assumption meets naive realism, guess what usually wins? It ain’t the truth.
At the risk of misusing the word, I’ll just point out something about our “truth”: that we all filter everything we learn through the structure of our own beliefs and the mental models we construct to support those beliefs. I like to think of science as a powerful means of penetrating the structure of those mental models, but that’s probably not a good analogy. That’s because, for science to work at changing our preconceptions, we have to have the validity of science already strongly incorporated into the structure of our own mental models. If it’s not, then science is more likely to bounce harmlessly off of the force field our beliefs create to repel it. (Sorry, I couldn’t help it; I’m a hopeless geek.) As a result, all other things being equal, when people see studies that confirm their beliefs they tend to view them as unbiased and well-designed, while if a study’s conclusions contradict a person’s beliefs that person is likely to see the study as biased and/or poorly done. As MacCoun puts it, “If a researcher produces a finding that confirms what I already believe, then of course it’s correct. Conversely, when we encounter a finding we don’t like, we have a need to explain it away.”
There’s also another strategy that people use to dismiss science that doesn’t conform to their beliefs. I hadn’t thought of this one before, but it seems obvious in retrospect after I encountered a recent study that suggested it. That mechanism is to start to lose faith in science itself as a means of making sense of nature and the world. The study was by Geoffrey D. Munro of Towson University in Maryland and appeared in the Journal of Applied Social Psychology under the title of The Scientific Impotence Excuse: Discounting Belief-Threatening Scientific Abstracts.
There were two main hypotheses and two studies included within this overall study. Basically, the hypothesis was that encountering evidence that conflicts with one’s beliefs system would tend to make the subject move toward a belief that science can’t study the hypothesis under consideration, a hypothesis known as the “scientific impotence” hypothesis or method. In essence, science is dismissed as “impotent” to study the issue where belief conflicts with evidence, thus allowing a person to dismiss the science that would tend to refute a strongly held belief. The problem, of course, is that the major side effect of asserting scientific impotence discounting is that it leads a person to distrust all science in general, or at least far more science than the science opposing that person’s belief.
Munro makes the implication of scientific impotence discounting plain:
The scientific impotence method of discounting scientific research that disconfirms a belief is certainly worrisome to scientists who tout the importance of objectivity. Even more worrisome, however, is the possibility that scientific impotence discounting might generalize beyond a specific topic to which a person has strong beliefs. In other words, once a person engages in the scientific impotence discounting process, does this erode the belief that scientific methods can answer any question? From the standpoint of the theory of cognitive dissonance (Festinger, 1957), the answer to this question could very well be “Yes.”
Not surprising, the scientific impotence discounting strategy of denying science permits one to dodge the charge of hypocrisy:
Using the scientific impotence excuse for one and only one topic as a result of exposure to belief-disconfirming information about that topic might put the individual at risk for having to acknowledge that the system of beliefs is somewhat biased and possibly hypocritical. Thus, to avoid this negative self-view, the person might arrive at the more consistent — and seemingly less biased — argument that science is impotent to address a variety of topics, one of which happens to be the topic in question.
To test these hypotheses, basically Munro had a group of students recruited for his study read various abstracts (created by investigators) that confirmed or challenged their beliefs regarding homosexuality and whether homosexuality predisposes to mental illness. It turned out that those who read belief-challenging abstracts were more prone to use scientific impotence discounting as an excuse to reject the science, while those who read belief-confirming abstracts were less likely to subscribe to the scientific impotence excuse. Controls that substituted other terms for “homosexual” demonstrated that it was the belief-disconfirming nature of the abstracts that was associated with use of scientific impotence discounting as a reason to reject the conclusions of the abstract. A second study followed up and examined more subjects. The methodology was the same as the first study, except that there were additional measures performed to see if exposure to belief-disconfirming abstracts was associated with generalization of belief in scientific impotence.
In essence, Munro found that, relative to those reading belief-confirming abstracts, participants reading belief-disconfirming abstracts indicated more belief that the topic they were reading about could not be studied scientifically and more belief that a series of other unrelated topics also could not be studied scientifically. In other words, scientific impotence discounting appears to represent a generalization of discounting of science from just science that challenges a person’s beliefs to more areas of science, if not all science. Munro concluded that being presented with belief-disconfirming scientific evidence may lead to an erosion of belief in the efficacy of scientific methods, also noting:
A number of scientific issues (e.g., global warming, evolution, stem-cell research) have extended beyond the scientific laboratories and academic journals and into the cultural consciousness. Because of their divisive and politicized nature, scientific conclusions that might inform these issues are often met with resistance by partisans on one side or the other. That is, when one has strong beliefs about such topics, scientific conclusions that are inconsistent with the beliefs may have no impact in altering those beliefs. In fact, scientific conclusions that are inconsistent with strong beliefs may even reduce one’s confidence in the scientific process more generally. Thus, in addition to the ongoing focus on creating and improving techniques that would improve understanding of the scientific process among schoolchildren, college students, and the general population, some attention should also be given to understanding how misconceptions about science are the result of belief-resistance processes and developing techniques that might short-circuit these processes.
On a strictly anecdotal level, I’ve seen this time and time again in the alt-med movement. A particularly good example is homeopathy. How many times have we seen homeopaths, when confronted with scientific evidence finding that their magic water is no more effective at anything than a placebo, claiming that their magic cannot be evaluated by randomized, double-blind clinical trials (RCTs)? The excuses are legion: RCTs are too regimented; they don’t take into account the “individualization” of homeopathic treatment; unblinded “pragmatic” trials are better; or the homeopaths’ anecdotal evidence trumps RCT evidence. Believers in alt-med then often generalize this scientific impotence discounting to many other areas of woo, claiming, for example, that science can’t adequately measure that magical mystical life energy field known as qi or even, most incredibly, that subjecting their woo to science will guarantee it to fail because belief is required and skepticism results in “negative energy.” Another common strategy I’ve seen for scientific impotence discounting is to dismiss science as “just another religion,” just as valid as whatever woo science is refuting, or to label science as “just another belief system,” as valid as any other. In other words, postmodernism!
Sadly, though, even physicians ostensibly dedicated to science-based medicine all too easily fall prey to this fallacy, although they usually don’t dismiss science as being inadequate or unable to study the question in question. Rather, they wield their preexisting belief systems and mental frameworks like a talisman to protect them from having to let disconfirming data force them to change their beliefs. Alternatively, they dismiss science itself as “just another belief.” Perhaps the most egregious example I’ve seen of this in a long time occurred, not surprisingly, over the mammogram debate from six months ago, when Dr. John Lewin, a breast imaging specialist from Diversified Radiology of Colorado and medical director of the Rose Breast Center in Denver, so infamously said, “Just the way there are Democrats and Republicans, there are people who are against mammography. They aren’t evil people. They really believe that mammography is not important.”
Wow! A straw man argument (that those who support the USPSTF guidelines are “against mammography”) combined with likening science to just another political viewpoint, and a condescending disclaimer that those who disagree with him “aren’t evil”! Mike Adams couldn’t have said it better. I wonder if Dr. Lewin thinks that Dr. Susan Love is “against mammography,” given that in the very same article it was pointed out that Dr. Love supports the USPSTF guidelines.
I get it. I really do. I get how hard it is to change one’s views. I even understand the tendency to dismiss disconfirming evidence. What I like to think distinguishes me from pseudoscientists is that I do change my mind on scientific issues as the evidence merits. Perhaps the best example of this is the aforementioned USPSTF mammography screening kerfuffle. For the longest time, I agreed enthusiastically with the prevailing medical opinion that screening for breast cancer with mammography beginning at age 40 was almost completely a universal good. Then, over the last two or three years, I’ve become increasingly aware of the problem of lead time and length bias, the Will Rogers effect, and overdiagnosis. This has led me to adjust my views about screening mammography. I haven’t adjusted them all the way to the USPSTF recommendations, but I am much more open to changes in the guidelines published late last year, even to the point that encountering the resistance of my colleagues led me to feel as though I were an anomaly.
Skepticism and science are hard in that they tend to go against some of the most deeply ingrained human traits there are, in particular the need for certainty and an intolerance of ambiguity. Also in play is our tendency to cling to our beliefs, no matter what, as though having to change our beliefs somehow devalues or dishonors us. Skepticism, critical thinking, and science can help us overcome these tendencies, but it’s difficult. Perhaps that’s the most important contribution of the scientific method. It creates a structure that allows us to change our beliefs about the world based on evidence and experimentation without the absolute necessity of taking being proven wrong personally.
When the scientific method is really embedded in the culture of scientists, it leads to the sorts of behavior described by Richard Dawkins where a scientist gives a talk with very solid evidence supporting his conclusions. That evidence, it just so happened, completely disconfirmed the long-held hypothesis championed by a very senior and respected member of his department. When the talk was over, everyone waited to see what this senior scientist would say. Instead of challenging the speaker, the scientist got up and thanked him for having shown him that he had been wrong for these many years because a phenomenon he was interested in was now better understood. True, the story may have been apocryphal or exaggerated, but that is the ideal of science. It is an ideal that is very hard, even for scientists, to live up to, and it’s even harder for non-scientists even to understand.
In the end, though, we need to strive to live up to the immortal words of Tim Minchin when describing how he’d change his mind about even homeopathy if presented with adequate evidence. Actually, as much as I love Tim Minchin’s humorous take on dealing with a woman espousing a panoply of woo that would rival Whale.to’s collection, I’m forced to realize that Minchin is a bit too flippant about the difficulty in changing one’s mind. I know, I know, he’s a comic musician (or a musical comedian); flippancy is part of his job. Even so, show me, for example, strong evidence that vaccines are associated with autistic regression, and I might not “spin on a dime” and change my beliefs, as Minchin put it, but eventually, if the evidence is of a quality and quantity to cast serious doubt on the existing scientific evidence that does not support a vaccine-autism link, I will adjust my views to fit the evidence and science. I’m also under no illusion about how difficult that would be to do or that I might even be prone to using some of the defense mechanisms described by psychologists to avoid doing that, at least at first. But I’ve changed my mind before based on science disconfirming medical dogma that I had long believed on more than one occasion. I can (and no doubt will) do it again.
In the end, I want knowledge, both to provide the best care possible for my patients and just for knowledge’s sake. Science is the best way to get that knowledge about the natural world, and no other methodology has improved the treatment of human disease as fast as science-based medicine. The price is one that I’m willing to pay: Uncertainty and the expectation that much of what I “know” now will one day have to change. What they told me my first day in medical school really is true. Twenty years after I graduated, at least half of what I was taught in medical school has changed or is no longer applicable to patient care, and that’s a good thing.
Certainty is nice, but I’ve learned to live without it.
REFERENCE:
Munro, G. (2010). The Scientific Impotence Excuse: Discounting Belief-Threatening Scientific Abstracts Journal of Applied Social Psychology, 40 (3), 579-600 DOI: 10.1111/j.1559-1816.2010.00588.x

Uncertainty in Medicine
Harriet Hall on May 24, 2016
Link
Medicine is an uncertain business. It is an applied science, applying the results of basic science knowledge and clinical studies to patients who are individuals with differing heredity, environment, and history. It is commonly assumed that modern science-based doctors know what they are doing, but quite often they don’t know for certain. Different doctors interpret the same evidence differently; there is uncertainty about how valid the studies’ conclusions are and there is still considerable uncertainty and disagreement about things like guidelines for screening mammography and statin prescriptions.
Snowball in a Blizzard by Steven Hatch, MD, is a book about uncertainty in medicine. The title refers to the difficulty of interpreting a mammogram, trying to pick out the shadows that signify cancer from a veritable blizzard of similar shadows.
Dr. Hatch starts with a clinical vignette. An elderly woman is in the ICU, and doctors on rounds are discussing her case and debating how to proceed. Her daughter asks, “So, you don’t know why she’s sick?” The doctor says he isn’t sure. “Do you think she should get a CAT scan?” He says he’s not sure; it would help pin down a diagnosis, but there is a risk that the contrast material might damage her ailing kidneys, perhaps irreversibly. “Do you think she needs antibiotics?” “Yes, I’m pretty sure, at least until we have another explanation that would indicate we can safely stop them.” As with practically everything in medicine, it’s complicated.
Dr. Hatch says:
Uncertainty lies at the heart of what physicians do on a daily basis. Sometimes they are entirely aware of it, and sometimes they fail to appreciate it. Sometimes it prominently features in discussions between doctor and patient. And sometimes it is completely misunderstood. The purpose of this book is to show the reader not only that this is so, but how it is so as well.
And he does a superb job. He provides numerous examples and clear explanations, covering many of the topics we have addressed on Science-Based Medicine. Very little in his book will be new to our regular readers, but he ties a lot of information together into a convenient package. He explains the many ways in which clinical research can yield misleading results, the difference between absolute and relative risk, the difference between correlation and causation, the number needed to treat (NNT), predictive value, the process of developing treatment guidelines, how the precision of new diagnostic technology is often mistaken for certainty, the dangers of false positive tests, the inability of doctors to predict how long an individual patient has to live, the hazards of unnecessary tests and overdiagnosis, how evolution prepared us to find narrative more compelling than statistics, confirmation bias, how our skill at pattern recognition makes us see patterns that are not real, pseudodiseases, the rationale for controlled trials with randomization and blinding, media malfeasance, illogical conspiracy theories, and much more.
He boils down the things we know for certain about maintaining our health to three principles: exercise more, eat less, and do not smoke. “Everything else is commentary.”
He provides a wealth of examples:
    • The Rosenhan experiment where fake patients were treated as schizophrenics, highlighting the uncertainty of psychiatric diagnosis.
    • PSA screening: the debate boils down to one life saved for every 1,000 men screened over 10 years versus many lives impacted by invasive procedures and unnecessary treatment.
    • Mammography: the data on the balance of benefits and harms for women under the age of 50, and how politics and misunderstandings made people question the unanimous consensus guidelines of experts.
    • Blood pressure treatment: how uncertainty led to genuine controversy among the experts.
    • Chronic Lyme Disease (CLD): it probably isn’t real, and there is clear evidence that prolonged antibiotic treatment does no good. The experts agreed unanimously, but a group of CLD advocates influenced public opinion. An independent panel “whose scientific integrity and objectivity was deemed unimpeachable by both parties” agreed with the scientific consensus. But the CLD advocates still managed to get legislation passed, using government bureaucracy to overthrow the scientific findings. Hatch tells the depressing story in all its excruciating detail.
    • Controversies over statins, antidepressants, and blood pressure drugs: he explains why assessing the benefit/risk ratio is tricky. The benefits are greater for the more severely affected patients. Lowering systolic blood pressure from 170 to 140 clearly saves lives; lowering it from 150 to 140, not so much.
    • How a treatment with uncertain benefits (DES to prevent miscarriage) was found to probably cause cancer in the children of mothers who took it during pregnancy.
    • The hormone replacement debate and how the study results were misrepresented to the public, with media-induced hysteria (“if you value your life, don’t even be in the same room as a bottle of hormones.”)
    • The HPV vaccine manufactroversy, where the media imagined a controversy that didn’t exist, based on misleading reports of adverse effects and deaths from the VAERS database. He points out that when you do anything to 10 million people, even give them a glass of water, some of those people will develop serious medical problems (that they were going to develop anyway).
Conclusion: Uncertainty and hope
There is a lot of discouraging news here, but there is also hope. By understanding the principles in this book, patients can better judge the uncertainties of what they hear. He praises HealthNewsReview.org, a website that can help consumers make sense out of media reports. They found that one in three news stories is a mere regurgitation of a press release that lacks critical information. He doesn’t mention Science-Based Medicine, which we think is another valuable source of reliable information; but he does cite Orac on page 199. Medicine is full of uncertainties, but doctors and patients can work in partnership to understand where a diagnosis or treatment falls on the uncertainty spectrum and that information can help them make rational decisions for individual health care.

The Fog of Medicine
Mark Crislip on January 22, 2016
Link


I often get called on to be a diagnostician. The referring doctor is uncertain what is going on in the patient, often a fever of unknown origin, and they call me to help figure it out. Sometimes I do, sometimes I don’t.
Making the correct diagnosis is not easy, even after 35 years. The classic phrase is the fog of war, but the fog of medicine is equally confusing. In retrospect sometimes a diagnosis becomes clearer, but in real time? It is so easy to be wrong and so difficult to be correct.
I remember the first case of my career. A patient developed neutropenia (low white count) from a sulfa antibiotic and over the next two weeks went into multi-organ system failure and died. At autopsy it was discovered that he had miliary tuberculosis. I totally missed the diagnosis, despite all the tests, including liver and bone marrow biopsies before he died.
A decade later a case was presented at conference about a patient with a fever who went into multi-organ system failure and died. At the time of the conference I had zero recollection of the case from a decade earlier, but knew this was a case of miliary TB based on the data presented, although it was far from a textbook case. After the conference the presenter let me know that it had been my case, the one I had missed years ago.
I was ten years wiser in my medical career and more aware of the subtleties and variations of disease presentation. Experience has made me a somewhat better diagnostician. Or so I hope. Making the diagnosis of a common presentation of a common disease should be simple, but patients rarely read the textbooks and we are more likely to see an uncommon presentation of a common disease, a common presentation of an uncommon disease, and most dreaded, an uncommon presentation of an uncommon disease.
I still have difficulty making the correct/accurate diagnosis on patients and so often I am left with best probability based on the often incomplete and contradictory data. I could make an accurate diagnosis on every patient, but for some reason they are hesitant to allow an autopsy while still alive. Go figure.
I kill me, hopefully not patients
Being in a teaching hospital not only do I need to teach the residents about the diseases I see, but how I get the diagnosis. I think my time in the skeptical world has made me a better physician as I try to be aware the various cognitive biases that make us poor thinkers. I like to use “Observations on spiraling empiricism: its causes, allure, and perils, with particular reference to antibiotic therapy” as a good Infectious Disease example of cognitive biases and try and apply its lessons to myself and my teaching.
There has been a recent focus on diagnostic error by physicians as a source of patient harm and the numbers are remarkable.
About 5 percent of adults who seek outpatient care annually suffer a delayed or wrong diagnosis.
Postmortem research suggests that diagnostic errors are implicated in one of every 10 patient deaths. Not every death is scrutinized, however, so the findings can’t be generalized to all hospital patients.
Chart reviews indicate that diagnostic errors account for up to 17 percent of hospital adverse events.
Diagnostic errors are the principle cause of paid malpractice claims and are almost twice as likely to end in a patient’s death than claims for other medical mishaps. They also represent the biggest share of total payments.
Those numbers do not surprise me. As Barbie said, medicine is hard. There are systems reasons for making the wrong diagnosis that boil down to too much data and too little time to process it. The use of Electronic Medical Records was supposed to help solve some of these issues. Hahahahahahahahahahahahahahahahahahahaahahahahahaha. Wipes eyes. I kill me sometimes.
There are knowledge deficits; we tend to diagnose what we know and miss the diseases of which we are unaware. Knowledge deficits are correctable with education. Hahahahahahahahahahahahahahahahahahahaahahahahahaha. Wipes eyes. I kill me again.
I once had a case of a young female with fevers, severe joint and muscle pains, high white count and a sed rate of 120 who I saw in clinic as an add on at the end of a Friday. She was sick, but I had no idea with what. I thought maybe endocarditis, so I admitted her and left for the weekend. When I returned on Monday she had been discharged. What did she have? I asked the hospitalist. Oh, she said nonchalantly, as if it were oh so obvious, it was Still’s. And upon review it was a textbook case. If I had ever read the textbook, I didn’t remember it. Totally missed the diagnosis. And even more annoying is I missed another case about three months later. Needless to say now I always consider Still’s. But one of the odd medical phenomena is how often you see your first case of a disease shortly after you learn about it. I so hate the Baader-Meinhof Phenomenon. But it makes me wonder how many similar cases I had missed in the past.
But the why and how of misdiagnosis are, I suspect, much deeper than lack of knowledge and systems problems. As skeptics know all too well, humans are fundamentally poor thinkers. We have evolved to survive reality, not to understand it, and the biggest problem with making the correct diagnosis is probably being human. I think the IOM is going to have their work cut out for them.
Last week I attended a Grand Rounds that was an excellent review of the cognitive biases as applied to medical diagnoses. No new cognitive biases were mentioned, but it was interesting to hear them in the context of medical diagnosis.
One nice review has 32 biases as applied to medical decision making. All of them are medical examples of cognitive errors and shortcuts we all use every day. The issue, of course, is how to get physicians to not only recognize the biases, but recognize that the biases apply to them. And there is the impenetrable reality distortion field generated by the Dunning–Kruger effect to contend with.
Solutions?
Perhaps more AI computer support. I Google/Pubmed everything I see and I often get ideas and directions I would not otherwise have considered. Perhaps crowdsourcing diagnoses will help. Perhaps edumatcating physicians on the cognitive biases. The best bet is to use CRISPR technology to build humans with improved cognitive abilities, the first step to the Zombie Super Soldier Army I so want to control. Or is that just me?
The problem is that cognitive error is high-hanging fruit and difficult to get at, and there will be a tendency to pursue more readily attainable goals…Real solutions to cognitive diagnostic errors lie in the shadows, and they will be difficult to find.
As I say medicine is hard and solutions to some problems may lie at the heart of being human.
Complementary and alternative medicine, and poor decision-making
That’s medicine. How about SCAM? How does one reduce diagnostic errors when the whole field of practice is fundamentally based on the unknowing application of every cognitive bias? In medicine we see the 32 biases of clinical decision making as a problem. SCAM must deny them to exist. To recognize them as applicable means the end.
The alternative medicine world is notorious for how multiple practitioners are unable to reach the same diagnosis when presented with the same patient. TCM practitioners can’t agree. Chiropractors can’t agree. Acupuncturists can’t agree. And naturopaths, proudly having no standard of care, can’t agree.
The entire field of Integrative/Alternative/Complementary Medicine is based on making the wrong, often imaginary, diagnosis then running with it. No need to worry about cognitive biases in world that is the medical equivalent of Lake Wobegon:
where all the diagnoses are correct, all the therapies are effective, and all the providers are above average.
Medicine may be plagued by misdiagnosis and its consequences, but we have some hope that these issues can be corrected or ameliorated.
SCAM is defined by misdiagnosis and its consequences.
I wonder if the IOM will ever tackle that problem.

The Semmelweis gambit: A red flag for defending bad science and quackery
A proponent of the Great Barrington Declaration is comparing rejection of its “natural herd immunity” approach to the pandemic to the rejection of Ignaz Semmelweis and his findings. It’s a deceptive comparison beloved of all manner of scientific cranks.
David Gorski on January 6, 2025
Link
Last week, SBM blogger Dr. Jonathan Howard took note of a post on (not-so) Sensible Medicine by Edward H. Livingston, MD, FACS entitled The Semmelweis Effect and The Great Barrington Declaration. Dr. Howard pointed out, quite correctly, that Dr. Livingston appears not to have had clue one about what the Great Barrington Declaration actually said and proposed. Why do we say this? In his post, Dr. Livingston never even mentioned that the central concept behind the GBD was to stop all those nonpharmaceutical interventions (NPIs), such as school and business closures, being used to slow the spread of the virus in favor of “opening up” society and allowing mass infection of the “young and healthy,” who were presumably at low risk for severe disease and death from COVID-19. The reason was, according to the GBD authors, that doing so would allow us to reach “natural herd immunity” more rapidly; indeed, they promised “natural herd immunity” in 3-6 months if their recommendations were followed. But what about those who were at high risk of severe disease and death from COVID-19, such as the elderly and those with various chronic diseases that made them high risk for complications and death? The GBD proposed “focused protection” for them without ever actually proposing any unique, concrete, practical methods to actually protect the vulnerable. Remember, also, that the GBD was published in early October 2020, which was more than two months before the mRNA vaccines became available to healthcare workers under an emergency use authorization in December 2020. (I got my first dose on December 18, 2020, the first week the new vaccines were available in my area.) Truly, the GBD was profoundly social Darwinist at its core.
Of course, the GBD was never a serious policy proposal. Rather, it was propaganda, a list of demands, custom-designed to provide a scapegoat for failures of pandemic policy. Besides violating basic principles of public health in a pandemic, the GBD also never would have worked, as a flack at the Brownstone Institute (the “spiritual child of the GBD”) tacitly admitted three years ago, and it arguably did great harm to public health in a number of countries. Unfortunately, one of the three authors of the GBD, Stanford health economist Dr. Jay Bhattacharya, is now poised to become the Director of the National Institutes of Health, thanks to his being nominated by President-Elect Donald Trump, likely at the behest of Robert F. Kennedy, Jr. As a result, his apologists are frantically trying to gaslight you about his history, such as when he and fellow GBD co-author Martin Kulldorff appeared at an antivax conference in 2022 organized by one of the most ridiculously silly and deranged antivaxxers of all, Steve Kirsch.
May 2022: That’s Jay Bhattacharya on the far left, with Martin Kulldorff sitting next to him, and Steve Kirsch next to him pontificating how COVID-19 vaccines have supposedly killed a half a million people. Bhattacharya could play with his phone all he wanted, and Kulldorff could try to lean back so far in his chair that he is almost invisible to the audience, but they were there helping Kirsch amplify his false claim based on an Internet survey on his Substack that COVID-19 vaccines had killed a half a million people.
As you might imagine, it’s not just enough to gaslight. No. Bhattacharya’s defenders are now going out of their way to represent him as a brave maverick, as someone whose ideas were “proven right” (as evidenced by his nomination), as a rejected, ostracized genius on par with—you guessed it!—Ignaz Semmelweis, and Dr. Livingston embraces this comparison whole-heartedly. Even though Dr. Howard has discussed the largest glaring error in Dr. Livingston’s post, namely his apparently ignorance about what the GBD actually said in favor of declaring the GBD (and thus Dr. Bhattacharya) as having been “vindicated” after supposedly small-minded attacks by dogmatists who were threatened by his idea, I thought I’d discuss this as well, but from (hopefully) an angle sufficiently different that you don’t view my discussion as a retread. Basically, I want to discuss this as part of the seemingly desperate desire that cranks have to be seen as “Semmelweis” or a “persecuted genius” who one day will be vindicated.
Let’s start with Dr. Livingston’s comparison.
Invoking Ignaz Semmelweis
Dr. Livingston sets the stage by explicitly likening Semmelweis’ discovery to the GBD. Subtle, he is not:
The story is often told in medical school about Ignaz Semmelweis, the Hungarian obstetrician who discovered that hand washing with chlorine reduced maternal mortality from puerperal fever. An intervention seemingly so simple and obvious but when first proposed in the 1840s, the experts of the day not only rejected it, but condemned Semmelweis as a heretic. The parallels between Semmelweis’s story and the Great Barrington Declaration (GBD) authors recall the adage that history may not repeat itself, but it sure rhymes.
Here’s the thing that’s often forgotten about this story, namely that it was not so obvious at the time that this intervention should work. I’ll elaborate, but first let’s see where Livingston goes with this comparison:
In the 1840’s, Semmelweis was the equivalent of a modern-day Chief Resident (sans work hours limitations) on an obstetric ward at Vienna General Hospital. Physician-led wards had much higher puerperal fever rates than did the adjacent wards staffed by midwives.
Autopsies were common back then. Physicians, not midwives, attended autopsies and rotated frequently throughout the day between the autopsy and delivery rooms. A friend of Semmelweis cut his finger during an autopsy. He developed sepsis very similar to puerperal fever and died. Given the similarities between the way his friend died and death from puerperal fever, Semmelweis thought that there must be some pathogen transmitted by dead tissue causing sepsis. He noted the smell of death was on his hands after performing an autopsy and that this smell did not do away with conventional hand washing with soap and water. The odor could be eliminated with chlorine. Semmelweis then adopted a chorine-based hand washing regimen resulting in a marked decrease in puerperal fever on his ward.
Semmelweis spent the next 20 years trying to convince the obstetric community of the benefits of chlorinated water washing. He was ignored. Luminaries of his time such as Virchow, of node fame, chastised Semmelweis believing that puerperal fever was a manifestation of venous thrombosis caused by abnormal uterine contractions. It is said that Semmelweis went insane because his life-saving discovery was rejected by the academic community. He died in an asylum.
That is the conventional mythology of Ignaz Semmelweis. However, as is usually the case, history is not as clear, not as cut-and-dried, as represented by Dr. Livingston. Whenever “brave mavericks” try to claim that their ideas are being unfairly rejected by dogmatic scientists and physicians by invoking the story of how Ignaz Semmelweis’ results were rejected and he was ostracized by some physicians, I like to point out that his work preceded Louis Pasteur’s seminal work that led to the acceptance of germ theory by nearly two decades and Joseph Lister’s work by even longer. In other words, at the time Semmelweis did his initial study there was no theoretical mechanistic framework to explain why washing one’s hands with chlorine bleach, but not just soap and water, after doing autopsies could so dramatically decrease the risk of puerperal fever. (It’s not as though doctors didn’t wash their hands after doing autopsies back then.) In 2025, such an explanation seems brain-meltingly obvious. Not so much in 1847, when Semmelweis reported—but didn’t formally publish—his initial results), and, contrary to Dr. Livingston’s telling, Semmelweis did not propose a “pathogen.” Rather, he had to posit that unspecified “cadaverous particles” were responsible. He fortuitously did choose a solution of chlorinated lime (calcium hypochlorite, a form of bleach) for washing hands between autopsy work and the examination of patients mainly because it got rid of the smell from the cadavers, and fortunately it also was a sufficiently good disinfectant to eliminate most of the microorganisms causing puerperal fever. He had no way of knowing that that is what it did, though.
Moreover, Semmelweis wasn’t exactly correct; certainly he was wrong about a lot of things. While he was correct that washing with chlorine before attending childbirth would greatly decrease the risk of puerperal fever, he was wrong when he tried to explain why. For example, when it was observed that, even with the most meticulous chlorine handwashing, there was still a mortality rate from puerperal fever of about 1%, which Semmelweis explained by suggesting that self-infection took place, specifically that internally generated cadaveric particles were responsible, such as from tissue crushed in the birth process and eventually turning gangrenous. (We now know that, most likely, it’s bacteria that normally grow on the skin and in the urogenital tract that can cause this.) Semmelweis also thought that “cadaveric particles” were the primary cause of puerperal fever, even though it was (and is) bacteria, such as streptococcus infections, either type A, which is commonly found in the throat and nasopharynx of otherwise healthy carriers, or type B, which lives on the skin. Semmelweis also didn’t help himself by not publishing right away, although he and his students did write letters to directors of several prominent maternity clinics describing their recent observations. He helped himself even less later in his career by writing open angry letters to prominent European obstetricians, at times denouncing them as irresponsible murderers.
Interestingly, Semmelweis was not rejected everywhere. For example, after his lecture to the Medical Society of Vienna in 1850, documented by Dr. Heinrich Herzfelder, the First Secretary of the society:
Herzfelder also noted that Semmelweis’ views were opposed by Lumpe and Zipl, who argued that the data supported a miasmatic cause of childbed fever, but was supported by Chiari, Arneth, Helm, and Hayne.1,2 He concluded by saying that the position taken by Lumpe and Zipl, as well as by Scanzoni and Seyfert, were adequately refuted by Semmelweis’ solution to the problem, which, he said, “can be considered a triumph of medical research.”
As I’ve pointed out a number of times, though, none of this is to say that the story of Semmelweis is really that he brought all the criticism and ostracism on himself, revisionist history documented in the same article above notwithstanding. He did go against the prevailing scientific ideas of the day and did receive a lot of pushback, given that germ theory, which could have explained his results, was still at least a couple of decades away from widespread acceptance. His is indeed a cautionary tale of of how medical dogma can react to unexplained findings, leading some to call him, not Joseph Lister, the true father of asepsis and to speculate that if Semmelweis hadn’t died so young he would have lived to see his findings vindicated through the work of Louis Pasteur, Joseph Lister, and others.
Of course, the simple version of Semmelweis’ history fits Dr. Livingston’s narrative far better, leading him predictably (given my past experience with invocations of Semmelweis) to start recounting examples of what he considers the “Semmelweis effect” (more commonly known as the Semmelweis reflex), which involves the reflexive dismissal of ideas and evidence that go against current consensus. Nor would I discount the Semmelweis reflex as a major problem in the psychology of scientists. That being said, some of his examples are…questionable. For instance:
The problem of new ideas being rejected by experts is so pervasive it should have its own term, something like the Semmelweis Effect. The Semmelweis Effect originated in Vienna but about 100 years later in nearby Germany, history repeated itself. A German surgeon, Eric Muhe, developed a new technique for gall bladder removal in 1985 using telescopic instruments introduced into the abdomen through a series of small incisions – the first laparoscopic cholecystectomies. Muhe reported his first 100 cases to various German medical societies. Not only was his new approach to cholecystectomy rejected by his colleagues, but he was charged with murder in 1987 when one of his patients died of a complication unrelated to the laparoscopic approach procedure.
For many years, the French surgeon, Phillipe Mouret, was credited for inventing laparoscopic cholecystectomy a full two years after Muhe. Because Muhe was so thoroughly discredited by his colleagues for performing what the Germans at the time considered dangerous and irresponsible surgery, Muhe’s achievement went unrecognized for many years.
I lived through the transition from open cholecystectomy to laparoscopy cholecystectomy. Indeed, I was a resident at the time. When I went into the lab to do my PhD in 1990s, old-fashioned open cholecystectomy still reigned. When I came out of the lab in 1993, laparoscopic cholecystectomy had taken over, and I found myself embarrassed that, unlike other third year residents, I had no experience with the laparoscope or even just “driving the camera,” which is what the assistant does, keeping tings in view by moving the camera around as needed. If anything, laparoscopic cholecystectomy is a cautionary tale about the widespread adoption of a procedure before its safety was properly vetted in clinical trials. If you read about the history of laparoscopic cholecystectomy, you’ll find that its popularity far preceded the evidence for it. Its adoption was the very epitome of hype before evidence, with established surgeons frantically learning the procedure because, if they didn’t, their patients would go to surgeons who could do the procedure. Fortunately, evidence did eventually justify the superiority of the procedure, especially after the adoption of the critical view of safety, which greatly decreased the incidence of common bile duct injuries associated with the procedure, but it could have gone the other way. Odd, then, that Dr. Livingston compares Muhe to Semmelweis.
Dr. Livingston then rattles off some more questionable examples:
The Semmelweis Effect remains stubbornly common. The Effect was in play with PCR’s invention by Kary Mullis, discovery of the cholesterol receptor by Brown and Goldstein, with accepting that H. Pylori causes gastroduodenal ulcer disease (Barry Marshall), and identifying nutritional deficiencies as causing pellagra (Joseph Goldberger). The notion that tobacco smoking causes lung cancer was vigorously opposed by Sir Ronald Fisher, perhaps the most famous statistician, because of inconsistencies in the data supporting that hypothesis.
I’m not sure what the heck he is talking about regarding Kary Mullis; he was awarded a Nobel Prize with co-inventor Michael Smith for his invention of PCR. Barry Marshall is a commonly cited example as well, but it turns out that his “rejection” was very much overblown. (Funny that Dr. Livingston doesn’t mention Robin Warren, who worked with Marshall.) It’s true that Barry Marshall and Robin Warren first reported a curious finding of what they described as “unidentified curved bacilli on gastric epithelium in active chronic gastritis” (not an ulcer) in two letters to The Lancet, published on June 4, 1983. They reported that it wasn’t seen using traditional staining methods and suggested that they might be associated with gastritis. By 1992, multiple studies had been published establishing the causative role of H. pylori in peptic ulcer disease, and medical practice rapidly changed. That’s less than ten years, which, given how long it takes to organize and carry out clinical trials, is amazingly fast. Yet somehow a favorite denialist myth is that “dogmatic,” “close-minded” scientists refused to accept Marshall and Warren’s findings. It’s an example of a scientific consensus that deserved to be questioned, was questioned in the right way, and was overthrown—and rapidly, too. Although there was initial skepticism of their results, in actuality Marshall and Warren essentially won the day within a decade, by which time the standard of care for treating duodenal ulcers had evolved to using antibiotics. That’s actually an incredibly fast about-face for medicine, given how long it typically takes to bring an idea from bench to bedside.
As for Sir Ronald Fisher, WTF? Fisher was one of the few scientists by the late 1950s who had not yet been convinced that tobacco smoking played a major role in causing lung cancer, and, as described in this article, his work might well have been “influenced by personal and professional conflicts, by his work as a consultant to the tobacco industry, and by the fact that he was himself a smoker” but was also part of a more general methodological debate in statistics at the time regarding attributing causation. Ditto Joseph Goldberger, whose discovery that nutritional deficiencies caused pellagra came as a result of an experiment of highly questionable ethics, involving offering prisoners at the Rankin Prison Farm in Mississippi pardons for participating in a clinical trial testing a deficient diet, as well as another experiment involving the intentional attempt to infect volunteers without ethical review.
This brings me to the biggest “WTF?” part of Dr. Livingston’s invocation of Semmelweis, but first he can’t resist this metaphor:
Despite the rich history of the dangers of rejecting new or alternate ideas about disease, the medical establishment has not learned its lesson. With COVID, the Semmelweis Effect was in full force. This time, it was the GBD authors who were the witches burned at the stake.
“Brave mavericks” sure do love comparing science-based critics to superstition or religion, don’t they? Almost as much as they like to compare their “brave maverick” heroes to Semmelweis:
Jay Bhattacharya, who authored the declaration with Dr. Martin Kulldorff and Dr. Sunetra Gupta, is the face of the GBD and a modern day Semmelweis. 2 Bhattacharya and his colleagues issued the Great Barrington Declaration on October 4, 2020, the pandemic was raging and the response to it was chaotic, calling for a reasoned approach to COVID.
Seriously. I’m bored. Now here’s the “WTF?” comparison:
Early in the AIDS epidemic, CDC case officers realized that AIDS spread the same way as hepatitis. Because AIDS was not transmitted through the air, they knew it was safe to be in close proximity with the patients. All that was needed was the implementation of what we now know as universal precautions to avoid contact with blood or secretions. That AIDS patients did not need to be locked away was part of the message public health officials communicated. This was not easy given the public’s panic about AIDS in the 1980s. By the 1990’s, patients dying with AIDS were surrounded by loved ones.
An irony is why some of the same public health experts such as Dr. Anthony Fauci would reverse course during the COVID pandemic.
The basic approach that worked for AIDS should have been implemented for COVID. Understand its basic biology and then determine how to respond to the pandemic.
Never does it seem to occur to Dr. Livingston that AIDS and COVID-19 are very different viral diseases and that very different approaches to slowing their spread are indicated, while also conveniently ignoring the role of homophobia in contributing to the AIDS panic in the 1980s. He also seems to be asserting that the fear that HIV was spread through the air diminished because the CDC recognized that it was spread by bodily fluids. It didn’t, not for a long time anyway. That leads to my “WTF?” reaction to his comparison to COVID-19, which is spread by aerosolized respiratory secretions in the air. Indeed, if there is a real “Semmelweis reflex” moment regarding COVID-19, it’s how long health authorities continued to insist that COVID-19 was spread by large droplets, which don’t travel very far through the air, rather than aerosols, which can and do. Moreover, contrary to what Dr. Livingston would have you believe, scientists, physicians, and public health officials did craft recommendations based on the existing understanding of coronaviruses in 2020, of which SARS-CoV-2, the coronavirus that causes COVID-19, was but one. Seriously, Dr. Livingston’s analogy doesn’t even make any sense.
Nor does this bit of misdirection:
COVID is caused by the SARS CoV-2 virus, a coronavirus, the same family of viruses responsible for the common cold. What was known about disease caused by these viruses before the pandemic? From the 18th edition of Harrison’s Principles of Internal Medicine published in 2012 referring to the 2003 SARS :
“…The disease appeared to be somewhat milder in cases in the United States and was clearly less severe among children,” “…spread may occur by both large and small aerosols and perhaps by the fecal-oral route as well.” “…environmental sources, such as sewage or water, may also play a role in transmission.” “Some ill individuals (“superspreaders”) appeared to be hyper infectious and were capable of transmitting infection to 10-20 contacts…”
“…risk factors for severe disease include age>50 years and comorbidities such as cardiovascular diseases, diabetes, or hepatitis. Illness in pregnant women may be particularly severe, but SARS-CoV infection appears to be milder in children than in adults.”
One could copy these statements written about SARS in 2012 and apply them to the SARS-CoV-2 virus in 2024. The approach to COVID, proposed by the GBD, should have followed the same play book used by public health officials in the 1980s for AIDS. Take what is known about the virus, craft public policy from that knowledge and then educate the public about the disease in a way as to sooth their fears.
Funny, but Dr. Livingston doesn’t mention what health officials did to stop the spread of the original SARS. One notes that it’s widely accepted that SARS was an example of an epidemic that was contained largely through traditional public health interventions, such as index case identification, quarantines, and the like, not anything resembling what the GBD proposed. It’s hard for me not to suggest here that, had the original SARS been handled the way the GBD authors proposed handling SARS-CoV-2 and the COVID-19 epidemic, SARS would never have been contained and would have itself become a deadly pandemic, something that almost happened despite the effective public health interventions that ultimately contained it. Indeed, initial interventions for COVID-19 were explicitly based on what appeared to work for SARS, as described in an article from May 2020 in The Lancet entitled Can we contain the COVID-19 outbreak with the same measures as for SARS? The authors concluded:
Although there are striking similarities between SARS and COVID-19, the differences in the virus characteristics will ultimately determine whether the same measures for SARS will also be successful for COVID-19. COVID-19 differs from SARS in terms of infectious period, transmissibility, clinical severity, and extent of community spread. Even if traditional public health measures are not able to fully contain the outbreak of COVID-19, they will still be effective in reducing peak incidence and global deaths. Exportations to other countries need not result in rapid large-scale outbreaks, if countries have the political will to rapidly implement countermeasures.
Unfortunately, as we now know, most countries did not have this political will, and here we are.
Basically, the GBD went against everything learned from previous experience with respiratory viruses, including the 2002-2003 SARS outbreak, but Dr. Livingston would have you think that the GBD was The Real Science, with Jay Bhattacharya and his co-authors akin to a modern day Semmelweis and Anthony Fauci and all the others who criticized the GBD being akin to Semmelweis’s critics, when in fact the GBD violated many principles of public health science and what was known then about coronaviruses. The only reason the GBD seems to have been “vindicated” now is because of a persistent propaganda and misinformation campaign of revisionist history on the part of its advocates that forgets that the GBD was based on mass infection designed to reach “natural herd immunity” within 3-6 months and that it wasn’t just the claim that “lockdowns” do more harm than good.
Again, Dr. Livingston left out the core idea behind the GBD, as Dr. Howard pointed out on X, the hellsite formerly known as Twitter:
Here’s what @ehlJAMA censored. pic.twitter.com/TBP0gz5cDk
— Jonathan Howard MD (@19joho) December 31, 2024
No. 

Herd immunity via mass infection of unvaccinated people under 70 was the core of the the GBD and you didn’t even mention it. 

I’m not afraid to report what they actually said in 2020.

You hid this all from your readers.https://t.co/4nC1ypvJ9d
— Jonathan Howard MD (@19joho) January 1, 2025
Seriously, note how Dr. Livingston pivots to “lockdowns were harmful”:
The more important issue isn’t the immunity argument. The lockdowns were tremendously damaging to society, and the medical community has paid very little attention to that damage. Businesses were ruined, the education system was impaired, and we still don’t know the full ramifications of lockdowns.
Funny, that. Not so funny is how, ignoring the sustained campaign of disinformation on the part of GBD proponents and other COVID-19 minimizers and, later, antivaxxers, Dr. Livingston writes:
We now know that Dr. Bhattacharya and the other GBD authors were correct, and the experts were very wrong. The Semmelweis Effect. Going forward, experts should be careful before discrediting new ideas. The more passionate the experts are about countering something, the more skeptical the public will be about the experts.
We “know” nothing of the sort. This is an assertion without evidence, in actuality an assertion that contradicts available evidence. It was even known at the time that chasing “natural herd immunity” for COVID-19 was a pipe dream, as described by authors of the John Snow Memorandum. “Natural herd immunity” requires life-long—or at least long-lasting—postinfection immunity, and it’s long been known that postinfection immunity is relatively brief for coronaviruses, because coronaviruses are good at mutating to evade postinfection immunity that results from infection with prior variants. We saw this with the Delta and Omicron variants. Again, the GBD was not consistent with what we knew about coronaviruses, immunity, or public health, no matter how mightily Dr. Livingston strains to claim otherwise.
Seriously, I’m surprised that Dr. Livingston didn’t quote Arthur Schopenhauer’s famous (and wrong) adage, “All truth passes through three stages. First, it is ridiculed. Second, it is violently opposed. Third, it is accepted as being self-evident.” (Of note, Schopenhauer almost certainly never actually wrote or said this.) Besides being risibly wrong (such “truths” never get to the third stage because they’re not “truth”), Schopenhauer’s saying is a favorite of defenders of bad science everywhere. I’ll give Dr. Livingston credit for having restrained himself enough not to have used that common chestnut.
The appeal of the Semmelweis gambit
Many years ago I coined a term—at least, I think I was the first to coin it almost 20 years ago, although I can never by totally sure—the Galileo Gambit. The term was intended to describe the propensity of cranks, quacks, and advocates of pseudoscience like creationism to claim the mantle of Galileo (or Semmelweis), scientists who were at first rejected by the scientific orthodoxy of the time and had to fight to get their ideas accepted—or who did not live to see their ideas accepted, like Semmelweis. The implication, of course, is that their “unconventional” or “outside-the-mainstream” ideas, whatever they may be (alternative medicine, intelligent design, psychic abilities, etc.), are on the same plane as those of Galileo or Semmelweis and are being unfairly rejected by dogmatic scientists who are close-minded and unable to accept their genius. Sound familiar? I also said at the time that, frequently, defenders of such crankery will add a list of famous scientists or experts whose views were rejected by mainstream science but ultimately prevailed (just like Galileo or Semmelweis). Sound familiar? Given that Galileo is a rather poor example of this phenomenon given that the objection to Galileo’s heliocentric model were mainly religious and from the Catholic Church and that other astronomers were replicating his results, increasingly I’ve been thinking that I should stop calling this the Galileo gambit and start calling this the Semmelweis gambit.
As I pointed out at the time (and have pointed out many times since), obviously, this technique seeks to denigrate the experts who reject the crank claims as not knowing what they’re talking about or as close-minded, unable to have the vision that they do. It also deceptively tries to associate the quack, crank, pseudoscientist, or pseudohistorian with the theories and findings of great visionaries who went against conventional wisdom, were rejected by the experts of the day, and then later shown to have been correct. Although Michael Shermer has of late taken a heel turn, I have to concede that he was correct two decades ago when he said, “Heresy does not equal correctness.” Too bad he seems to have forgotten the wisdom of his earlier words, which is why I tend to like to quote Carl Sagan more often these days:
But the fact that some geniuses were laughed at does not imply that all who are laughed at are geniuses. They laughed at Columbus, they laughed at Fulton, they laughed at the Wright brothers. But they also laughed at Bozo the Clown.
Come to think of it, Columbus wasn’t the greatest example either. I might have to reformulate the idea that Sagan was expressing. Be that as it may, Dr. Livingston is far from the first defender of crank ideas to invoke Galileo or Semmelweis when he argued that the GBD was correct. Indeed, one of the GBD authors, Martin Kulldorff invoked not just Galileo but Tycho Brahe, Johannes Kepler, and René Descartes, likening the GBD (and the reaction to it) to the theories of those great scientists. Back in November 2020, Barry Brownstein also invoked Semmelweis defending the GBD and likened Sunetra Gupta (its third co-author) to him. It’s a common—and deceptive—theme that’s been used many times by GBD defenders, one of whom even more deceptively argued that “crushing scientific dissenters, as Fauci urges, would kill medical progress.” The common narrative is that the GBD’s authors are the “modern Semmelweis” and that “scientific dissent” is being crushed by modern-day versions of Rudolf Virchow, the most prominent critic of Semmelweis of the era. (Yes, I’ve seen Anthony Fauci likened to Virchow in this context.)
I’m not saying that when you see comparisons like Dr. Livingston’s to Semmelweis (or Galileo) that it’s always a sign that you’re dealing crankery, pseudoscience, quackery, or just plain bad science. After all, the oft-ignored complexities of his story notwithstanding, it is true that Semmelweis’s ideas were (mostly) rejected by his fellow physicians and scientists but then later shown to be true, at least true that washing hands with chlorine did dramatically decrease the risk of puerperal fever. It is true that, human nature being what it is, the Semmelweis reflex can be a real phenomenon that hinders the acceptance of new science and new medical treatments. However, often forgotten is that it can be very difficult to judge where the line between what would be reasonable skepticism given the knowledge of the time ended and the Semmelweis reflex begins. After all, science is about questioning and falsifying hypotheses, and at the time a new idea is being proposed it is not always easy to tell when questioning passes from reasonable to unreasonable. Moreover, as I like to point out, the vast majority of “heterodox” ideas are rejected precisely because they are wrong! Most such ideas fail the test of science, while only a relatively few stand the challenges of experimentation and evidence. Defenders of crank hypotheses or ideologically motivated bad science (like the GBD) cherry pick the few examples of real mavericks who were ultimately vindicated and ignore the much larger number of “brave mavericks” lost to the mists of history because their ideas never panned out.
Basically, it’s what I like to call the fallacy of future vindication, except that the proponents of the GBD, like Dr. Livingston, have leapt to declaring vindication now. Never once does the thought enter the minds of people like the authors of the GBD (or, frequently, their defenders like Dr. Livingston) that they might be wrong, that the reason they are viewed as cranks or treated with disdain is because they are so obviously wrong. That’s what separates them from real scientists. Sure, real scientists who believe unpopular things also believe that some day they will be vindicated, but they also carefully consider the possibility that they might actually be wrong and are prepared to change course if the evidence demands it. The authors of the GBD and their defenders fail that test.
Depressingly, Dr. Livingston Professor of Surgery at UCLA—it always depresses me when a fellow surgeon makes arguments this bad—and former Deputy Editor at JAMA. In a way, that’s a cautionary observation as well. Anyone can be prone to the Semmelweis gambit, if they believe in a bad idea strongly enough. At the risk of self-aggrandizement through humble-bragging, I like to think that the difference between me and someone like the GBD authors is that I always ask myself whenever I assert something whether I might be wrong and try hard to remain open to the possibility that I might be. I also know that my readers will disabuse me of such a notion, should I ever stumble as badly as Dr. Livingston has stumbled here. Drs. Jay Bhattacharya, Martin Kulldorff, and Sunetra Gupta are not akin to Ignaz Semmelweis, nor was the Great Barrington Declaration a discovery on par with antisepsis. Quite the contrary, two of the GBD authors sat at the same table as Steve Kirsch and the worst of the worst antivax cranks, and the GBD was not an idea that was ever vindicated.

   Postmodernist attacks on science-based medicine
David Gorski
Link
The postmodernist critique of science consists of two interrelated arguments, epistemological and ideological. Both are based on subjectivity. First, because of the subjectivity of the human object, anthropology, according to the epistemological argument cannot be a science; and in any event the subjectivity of the human subject precludes the possibility of science discovering objective truth. Second, since objectivity is an illusion, science according to the ideological argument, subverts oppressed groups, females, ethnics, third-world peoples.
-Melford Spiro, 1996.
Postmodernism and science
I detest postmodernism.
Well, it’s not really postmodernism per se that I detest. In the humanities, I don’t mind it so much, although reading postmodernist texts in college did make my head hurt. I suppose that in the humanities postmodernism provides a sometimes useful methodology for providing insights into interpretation of a wide variety of subjects in literature and the arts, although much of the time it seems to exist mainly to try to make texts mean exactly the opposite of what the words on the page say. Relying as it does on deconstruction, which is primarily a form of literary analysis, postmodern analysis is built on questioning the assumptions underlying any text, “deconstructing” its meaning. The problem is, it’s rare that a postmodernist critique of anything doesn’t consist of some of the densest, most impenetrable verbiage in existence.
Since I don’t claim to be a philosopher, and I haven’t studied postmodernism since college, I don’t intend to embarrass myself by trying to do a detailed–ahem–deconstruction of what postmodernism is. Many are the trees that have been killed to write books trying to explain what postmodernism is and how to apply it to various subjects. I do know, however, that postmodern philosphy is skeptical–even nihilistic–when it comes to the values of what is considered “modernity.” Unfortunately, to some, science is one of these modern values that is viewed with extreme skepticism, even to the point of representing the very essence of science as nothing more than one narrative among many. True, Spiro was writing primarily about anthropological science, but medicine is a “soft” enough science to be just as easily attacked in such a manner, and postmodernists don’t limit themselves to the “soft” sciences, anyway. Even worse, these sorts of arguments often claim that science (or, in this case, evidence-based medicine) is nothing more than a sort of hegemony of the power structure being imposed upon the very definition of “data” or “reality,” the implication that it’s us white males whose hegemony is being served (and whose hegemony, presumably, must be resisted) doing the imposing, as if there are no inherent characteristics in science that make it a more reliable means of assessing reality as it exists than, for example, personal anecdote and “experience.”
No wonder woo-meisters and those who hate the very concept of evidence-based medicine (as opposed to the deficiencies in how EBM ranks evidence, which in fact were the reason why this blog is called Science-Based Medicine and not Evidence-based Medcine) love postmodernism so much. It’s the perfect tool for them to appeal to other ways of knowing and try to make it seem as though scientific medicine is no more valid a construct to describe reality than that of the shaman who invokes incantations and prayers to heal, the homeopath who postulates “healing mechanisms” that blatantly contradict everything we know about multiple areas of science, or reiki practitioners who think they can redirect “life energy” (or qi) for therapeutic effect. In the postmodernist realm all are equally valid, as there is no solid reason to make distinctions between these competing “narratives” and the “narrative” of scientific or evidence-based medicine.
Perhaps the best quick explanation of how postmodernism is used to attack science comes from Rob Helpy-Chalk:
Knowledge was always in some way relativized to culture, so that it was possible to talk about many “equally valid ways of knowing” of which enlightenment science was only one. For instance, contemporary biologists say that the cassowary (an ostrich-like creature) is a bird, albeit one that cannot fly. The Karam people of New Guinea, who live alongside the cassowary, say that the cassowary does not belong in the same category as the birds (which they call yakt) but bats do belong to that category. So who’s to say that the biologists are right and the Karam are wrong? Knowledge is all relative.
Claims to knowledge were also always in some way “constructed” or “socially constructed” in the postmodernist movements. This meant that they had less to do with grasping the way the world actually works and more to do with creating social structures that advanced the interests of the people who claimed to have knowledge. The science of thermodynamics was not really a description of the properties of heat. It was about convincing people to buy steam engines and arranging society so that they would be happy when they bought one.
Thus, one of the key tenets of some strains postmodernism is that knowledge is relative and “socially constructed,” usually for the purpose of reinforcing the existing power structure or furthering the interests of the “elite” who have the knowledge. It is therefore not surprising that critics of evidence-based and science-based medicine would find postmodernism to be a very attractive philosophical mantle in which to wrap their objections to the science that does not support their favored understanding of how disease works or what treatments are effective. After all, if all knowledge is relative, then why shouldn’t their way of knowing be just as valid as that of science, given that in the postmodernist view as used (or, as many philosophers would characterize it, abused) by “postmodernist” critics, science is just “another narrative.”
Evidence-based medicine as “microfascism”
Perhaps the most hilariously over-the-top criticism of medicine as a Tool of the Man designed to suppress and control Brave Mavericks (not John McCain) was published two years ago in the form of an article by Dave Holmes, RN, PhD, Stuart J. Murray, PhD, Amélie Perron, RN, and Geneviève Rail, PhD, entitled Deconstructing the evidence-based discourse in health sciences: truth, power and fascism. The article was published, ironically enough, in the International Journal of Evidence-Based Health Care. I choose it for two reasons. First, it is so over-the-top that it brings the postmodernist attack on scientific medicine into sharp relief. Second, when you come right down to it, the sorts of arguments made by Holmes et al are the same as many arguments made by aficionados of complementary and alternative medicine (CAM).
Let’s look at the abstract:
Background: Drawing on the work of the late French philosophers Deleuze and Guattari, the objective of this paper is to demonstrate that the evidence-based movement in the health sciences is outrageously exclusionary and dangerously normative with regards to scientific knowledge. As such, we assert that the evidence-based movement in health sciences constitutes a good example of microfascism at play in the contemporary scientific arena.
Objective: The philosophical work of Deleuze and Guattari proves to be useful in showing how health sciences are colonised (territorialised) by an all-encompassing scientific research paradigm – that of post-positivism – but also and foremost in showing the process by which a dominant ideology comes to exclude alternative forms of knowledge, therefore acting as a fascist structure.
Conclusion: The Cochrane Group, among others, has created a hierarchy that has been endorsed by many academic institutions, and that serves to (re)produce the exclusion of certain forms of research. Because ‘regimes of truth’ such as the evidence-based movement currently enjoy a privileged status, scholars have not only a scientific duty, but also an ethical obligation to deconstruct these regimes of power.
I don’t know about you, but when it comes to my healthcare, I actually want my physicians to be “outrageously exclusionary”–exclusionary to quackery, that is. I also had no idea that the Cochrane Collaboration was so amazingly powerful. Whatever reasonable criticisms of the whole EBM paradigm may be, they don’t include labeling the Cochrane collaboration as fascistic. “Micro”-fascist that I apparently am for advocating science and evidence-based medicine, let’s see if I can place my “micro”-jackboot right on the throat of this sort of argument, apply sufficient pressure to make it gasp for air, and then apply my “micro”-truncheon liberally to its face, all the while resenting that the authors apparently didn’t consider EBM enough of a threat to label it real fascism. (I’ll leave it to the postmodernist authors of the paper to figure out whether I’m being facetiously metaphorical or not.)
Basically, this entire article is a huge appeal to other ways of knowing, coupled with massive straw men arguments, with a few other logical fallacies sprinkled in liberally for seasoning. The article begins:
We can already hear the objections. The term fascism represents an emotionally charged concept in both the political and religious arenas; it is the ugliest expression of life in the 20th century. Although it is associated with specific political systems, this fascism of the masses, as was practised by Hitler and Mussolini, has today been replaced by a system of microfascisms – polymorphous intolerances that are revealed in more subtle ways. Consequently, although the majority of the current manifestations of fascism are less brutal, they are nevertheless more pernicious.
You can almost hear the authors rubbing their hands together and cackling at being so…contrary, so…naughty, so…against the old stodgy scientists who think that we can actually know or measure reality with something close to objectivity. In other words, they’re just throwing in the comparison because it’s inflammatory. It’s tempting to appeal to Godwin’s law and just be done with it, but I think you’ll soon see why this attitude needs to be examined. They then state their objective:
Drawing in part on the work of the late French philosophers Deleuze and Guattari the objective of this paper is to demonstrate that the evidence-based movement in the health sciences is outrageously exclusionary and dangerously normative with regards to scientific knowledge. As such, we assert that the evidence-based movement in health sciences constitutes a good example of microfascism at play in the contemporary scientific arena.
Yes, I suppose that one would find EBM to be “outrageously exclusionary” if one is a quack. But let’s get to the meat (or what passes for meat in this article). First, here’s the straw man that the authors harp on over and over and over:
As a global term, EBHS (evidence-based health sciences) reflects clinical practice based on scientific inquiry. The premise is that if healthcare professionals perform an action, there should be evidence that the action will produce the desired outcomes. These outcomes are desirable because they are believed to be beneficial to patients. Evidence-based practice derives from the work of Archie Cochrane, who argued for randomised controlled trials (RCTs being the highest level of evidences) as a means of ensuring healthcare cost containment, among other reasons. In 1993, the Cochrane Collaboration, serving as an international research review board, was founded to provide clinicians with a resource aimed at increasing clinician-patient interaction time by facilitating clinicians’ access to valid research. The Cochrane database was established to provide this resource, and it comprises a collection of articles that have been selected according to specific criteria. For example, one of the requirements of the Cochrane database is that acceptable research must be based on the RCT design; all other research, which constitutes 98% of the literature, is deemed scientifically imperfect.
Not exactly. While it is true that the Cochrane Collaboration considers randomized clinical trials (preferably double-blinded) to be the a very highest form of evidence; it does not follow that all other research is “scientifically imperfect,” simply for the reason that RCTs are not considered “perfect.” No science is, nor does the Cochrane Collaboration insist on or use only RCTs in formulating its literature reviews. True, there are some “true believers” who do push rigid EBM as a panacea, but they are definitely in the minority and tend to be viewed skeptically among most other EBM advocates. Moreover, as this blog shows, if anything EBM undervalues certain forms of evidence, specifically basic science, thus making EBM problematic when studying non-science-based health care modalities. Nonetheless, other than its devaluing of basic science, EBM gets it pretty close to right when it comes to ranking forms of evidence derived from clinical trials.
What we’re really looking at here, I must emphasize again, is an appeal to a different way of knowing:
We believe that health sciences ought to promote pluralism – the acceptance of multiple points of view. However, EBHS does not allow pluralism, unless that pluralism is engineered by the Cochrane hierarchy itself. Such a hegemony makes inevitable the further ‘segmentation’ of knowledge (i.e. disallowing multiple epistemologies), and further marginalise many forms of knowing/knowledge.
Ah, yes. “Pluralism.” Whatever that means when it comes to scientific evidence. Does that mean that unscientific bunk like homeopathy should be considered as deserving of equal consideration as EBM, all in the name of “pluralism”? Personally, when it comes to pseudoscientific crap like this, EBM should be “outrageously exclusionary.” However, even more than the “exclusionary” nature of EBM, what really seems really to bother the authors is the language of EBM and science, presumably because of its emphasis on evidence, reason, and fact:
We believe that EBM, which saturates health sciences discourses, constitutes an ossified language that maps the landscape of the professional disciplines as a whole. Accordingly, we believe that a postmodernist critique of this prevailing mode of thinking is indispensable. Those who are wedded to the idea of ‘evidence’ in the health sciences maintain what is essentially a Newtonian, mechanistic world view: they tend to believe that reality is objective, which is to say that it exists, ‘out there’, absolutely independent of the human observer, and of the observer’s intentions and observations. They fondly point to ‘facts’, while they are forced to dismiss ‘values’ as somehow unscientific. For them, this reality (an ensemble of facts) corresponds to an objectively real and mechanical world. But this form of empiricism, we would argue, fetishises the object at the expense of the human subject, for whom this world has a vital significance and meaning in the first place. An evidence-based, empirical world view is dangerously reductive insofar as it negates the personal and interpersonal significance and meaning of a world that is first and foremost a relational world, and not a fixed set of objects, partes extra partes.
This is simply the age-old complaint that modern medicine doesn’t sufficiently value the humanity of the patient, that it’s cold and uncaring, interested only in reason and science, a complaint that has been around ever since we first started to base medical practice on science rather than folklore. I applaud these postmodernist authors, though, for showing admirable restraint in refraining from doing what I expected and mentioning Dr. Josef Mengele as a natural consequence of the fascistic scientific world view in which an objective reality is acknowledged. Besides, the universe doesn’t give a damn what we humans believe about it. Gravity will still cause you to tumble to the ground, whether you believe in it or not. Cancer cells don’t pay attention to the observer’s intentions and observations. Neither do microbes. As science has shown us, they do, however, pay attention to chemotherapeutics and antibiotics.
Indeed, the entire reason for the evolution of medicine from empirical, experience-based treatments to EBM comes from the very fact that the old ways were prone to a number of biases. The placebo effect guarantees that almost any intervention will make some proportion of patients with a given condition feel better, at least transiently. That’s the main reason for double-blinding and randomization when possible. Many diseases are either self-limited or their course waxes and wanes. If a patient tries something (or a doctor tries something on a patient) right before the disease symptoms wane, he or she will likely falsely attribute the clinical improvement to whatever was done right before, even though the intervention may or may not have had anything to do with the patient’s improvement! Confirmation bias means that, if you expect a treatment to work, your tendency, unintentional but real, will be to remember bits of evidence that fit with what you expect and to disregard or forget those that contradict what you expect. Because human observation is so fallable and so easily able to misrepresent objective reality is exactly what the scientific method corrects for. There are many other such pitfalls, thanks to human cognitive quirks and our brains’ hardwiring to attribute cause to correlation. It has been through experience, actually, that we have learned that the scientific method is the best method we currently have to correct for what would lead us astray.
Unfortunately, Holmes et al didn’t learn this lesson from all the justified criticism their article provoked. Apparently, now microfascism now requires microresistance.
On the nature of “evidence”
I’ll give Holmes and his crew of postmodernists credit for one thing. They don’t give up. Having been thoroughly slapped down by a large number of scientists for referring to EBM (and, by extension, science itself) as “microfascism,” they turn their postmodernist eye to evidence itself. In the very latest issue of the Journal of Research in Nursing, they have published a followup to their infamous “microfascist” paper entitled On the constitution and status of ‘evidence’ in the health sciences, this time first-authored by Stuart J. Murray. This article is almost as good (bad) as whole “microfascism” rant. (The only reason it’s not as good or better is the lack of a term as cringe-worthy as “microfascism.”) Basically, it’s the same tired old refrain that science is nothing more than the existing power structure exerting its will to enforce “conformity.” Because the article is not available to a lot of my readers, I feel obligated to quote fairly liberally from it to give you a flavor of just how ridiculous it is. The authors’ intent is made plain right from the first sentence:
What constitutes ‘evidence’ in the health sciences? Without too much hesitation, we might say that scientific evidence is worthwhile if it can be repeated, independently verified and measured according to standards upon which we can all agree. We might call this the ‘common sense’ view and commonsensically believe that these conditions hold as true not just in the sciences but also in our daily dealings in the world. We might even say that nothing could be more straightforward, that evidence in this context is obvious or ‘self-evident’ and that in the end, ‘seeing is believing’. In this paper, we suggest that this cheerful proverb stands as a kind of emblem for the dangerously naïve commonsense view on truth that has spread throughout our culture. We argue that this view betrays an almost unshakeable faith in the human capacity for unbiased or objective observation and analysis. Ultimately, this means that science becomes supplanted by ideology, and scientific inquiry becomes a ‘methodological fundamentalism’ (House, 2006).
This is of course, nothing but the old “fundamentalism” charge leveled against those of us who actually think that scientific rigor is important in determining what treatments work and what treatments do not. It’s the same charge routinely leveled against evolutionary biologists by creationists, be they “intelligent design” creationists or the old hard core young Earth creationists. It’s the same charge leveled by pseudoscientists of all stripes against scientists who correctly tell them they are pseudoscientists: That scientists are rigid, close-minded, unable to think “outside the box,” interested only in supporting existing understanding. To them, as I mentioned before, science is nothing more than a Tool of the Man to enforce cultural hegemony and, more importantly, to keep the Brave Mavericks down. In fact, the entire premise of this article is based on a huge straw man, as is shown here:
This paper discusses the constitution and status of ‘evidence’ in light of the almost evangelical rise of evidence-based health sciences (EBHS), including nursing. One salient example is the Cochrane Library, which provides EBHS acolytes with a database of ‘systematic reviews’ that has been faithfully constructed according to the evidencebased movement (EBM) directives. Thus, within the Cochrane Library’s hierarchy of allowable ‘evidence’, the randomised controlled trial (RCT) is taken as the ‘gold standard’, and proponents will scoff at any criticism because the RCT can be repeated, independently verified and measured according to standards upon which we can all (presumably) agree. It is not rocket science, or so they will say. But here with the seemingly innocent exaltation of the RCT, we find an explicitly hierarchical ranking that denigrates the evidentiary value of clinical experience; and similarly, qualitative research based, among other things, on participants’ narratives is ‘systematically’ ranked lower in value as ‘evidence.’
Thus, the most faithful proponents of EBHS must adopt a position in which “seeing is believing”–where evidence is presumed to be visual, immediate and incontestable. It is as if the evidence itself spoke the truth, and EBHS finally realised the dream of a pure science, a science free from the inherent messiness of human language, of human interpretation, of human values or, indeed, of anything recognisably human because the body that EBHS treats is the ‘average body’ generated by the RCT, without any experimental body. Nevertheless, within the culture of the health sciences, EBHS now circulates as a kind of fundamental truth, and it can do so because EBHS has come to control the terms by which evidence appears. The EBM constitutes a vast matrix of influence from funding bodies and academic institutions to nursing best-practice guidelines (BPGs) and multiple-related postulates that inform day-to-day heath care practices. In other words, for EBHS, ‘seeing is believing’ because EBHS carefully limits what can be seen in the first place. EBHS limits not only what can appear within our visual field but also how it will appear and how that evidence will be framed. EBHS appeals to a culture that is taught to embrace simple directives and to be suspicious of intellectual critique.
A more massive strawman I have a very hard time indeed imagining. Could someone tell me where in the Cochrane literature (or any EBM literature, for that matter–or any science literature, come to think of it) does it say that evidence must be “visual, immediate, and incontestable”? What a load of rubbish! If evidence were that obvious, then EBM wouldn’t even be necessary. Certainly that tool of EBM, the meta-analysis, would not be necessary because the very reason to perform a meta-analysis is to try to develop a consensus when there are multiple trials that do not all agree. In fact, EBM can be viewed as a means of trying to make sense of clinical data supporting various treatments, even when the data are not clear or are conflicting. “Seeing is believing?” Ha! If only EBM were that easy! Actually, if EBM were that easy, anyone could do it, while science itself should, in ideal circumstances, guide the way.
Murray and his crew then go on to claim that “seeing is not always believing,” taking arguments about the social sciences and invoking, of all things, the Rodney King trial and the videotape evidence used, concluding:
Our point in referring to what many saw as a manifest injustice is to call attention to our ways of seeing and the often hidden politics of the evidence to hand. In the King case, many of us would draw a very different conclusion from the visual ‘evidence’, even though it is exactly the same piece of evidence: it can be repeatedly screened, its accuracy can be independently verified and it can be measured according to standards upon which we can all (presumably) agree. Although the evidence seems to present a truth that is pure and simple, as Oscar Wilde once quipped, the truth is rarely pure and never simple. The evidence cannot simply speak for itself because the meaning of that evidence is of another order altogether. Thus, we must worry about the ways in which evidence is manipulated and contextualised under the aegis of efficiency, in the name of political expediency or in the name of scientific progress, and sometimes all three at once, as in the famous case of the Tuskegee Syphilis Study, to offer one tragic example.
A rather amazing line of reasoning, don’t you think? Because a jury interpreted a videotape in a way that didn’t make sense to most people, Murray seems to be arguing, that means that EBM–even science itself–is wrong, a false, politicized concept! Truly, you can’t make stuff like this up. At least, no one with a scientific outlook can, except perhaps as parody. Never mind that the jury of the Rodney King trial were not scientists, and it wasn’t the scientific method being applied. Indeed, one reason the scientific method exists is to serve as a check against the very sort of problems in human perception that allow verdicts like that of the Rodney King trial to occur. In fact, I’m surprised they didn’t bring up the O. J. Simpson verdict as well. And, of course, EBM is just like the Tuskegee Syphilis Study, if you’re to believe this. Never mind that in addition to being incredibly unethical the Tuskegee Syphilis Study was actually pretty dubious science given that much was already known about the progress of untreated syphilis at the time it was done, making it completely unnecessary. This entire argument is nothing more than taking a germ of a reasonable criticism of EBM and going right off the deep end with it:
Our critical perspective develops some of the five shortcomings of the EBM succinctly described by Cohen, et al. (2004), namely, that EBHS [1] relies too heavily on empiricism, [2] relies on too narrow a definition of evidence, [3] ironically, lacks any evidence of its own efficacy, [4] is of limited use for individual patients and [5] threatens the autonomy of the clinician or patient relationship. In addition, however, our work has attempted to politicise the ways of seeing that have become common in the health sciences as a result of the EBM agenda (Holmes, et al., 2008; Holmes, et al., 2006a, Holmes, et al., 2006b; Murray, et al., 2007). In other words, we have sought to examine, to bring to light, the unexamined postulates that underwrite the EBM. From this critical perspective regarding what is now the dominant episteme, we claimed that the EBM was politically dangerous (Holmes, et al., 2006a). Relying on the French philosophers Deleuze and Guattari (1987), we argued polemically that the EBM is akin to a totalitarian political structure and, consequently, that its way-of-seeing is informed by a politically dangerous ideology. In a nutshell, we noted that the EBM wholeheartedly adopts corporate models of efficiency and accountability, right down to a corporate lexicon; EBM relies reductively on quantitative evidence in which RCTs are fetishised; EBM denigrates other forms of knowledge, including clinician experience and patient testimony; finally, EBM evacuates the social and ethical responsibilities that ought to distinguish health care professions, such as nursing.
Oddly enough, I’d tend to agree with some of these. First, EBM does tend to rely too heavily on empiricism, but not in the way that is meant in the passage above. Our criticism of EBM is that EBM relies on empiricism while devaluing previous understanding of basic science that can guide us regarding the plausibility of proposed treatments. In other words, to EBM, it matters very little that, for example, homeopathy goes completely against the laws of chemistry and physics as we currently understand them, an understanding based on hundreds of years of research and understanding. All that really matters in EBM is what the clinical trials, especially the RCTs, say. While RCTs are the most rigorous way to test most medical hypotheses, that is only true if the hypotheses being tested have some measure of plasuibility based on basic and clinical science. Indeed, that is the very reason that EBM is so easily coopted by CAM advocates. RCTs testing hypotheses with low prior probability based on science inevitably generate false positive tests, the lower the prior probability the more likely the false positive. The answer to that problem is not to widen the definition of evidence; it is to narrow it further by explicitly taking into account basic science and prior probability in deciding which RCTs make sense to carry out. Prioritization with regards to whether an RCT should be done and in what order hypotheses should be tested by RCTs should be based on scientifically-determined estimated prior probability, which comes from cell culture, animal studies, and preliminary observations in humans. In other words, part of the answer is to become, in the terms of the “microfascism” article, even more “outrageously exclusive” in the sort of evidence we value as practitioners of science-based medicine–definitely not the kind of “solution” the authors would approve of. To us it’s about the science and using science to develop the best patient care. To postmodernist critic it’s not about evidence; it’s about resisting The Man, science be damned.
In fact, I suspect they’d consider Science-Based Medicine even more than “microfascism,” requiring “microresistance,” if this passage is any indication:
In short, we must find ways to combat the procrustean policies that have hijacked many modes of scientific inquiry and have led instead to a tangled web of ideological apparatuses, including Big Pharma; innumerable government lobbies; professional healthcare associations, such as the Registered Nurses’ Association of Ontario (Canada) and its compulsive endorsement of ‘best-practice guidelines’; academia and its research sponsors; the convergence of research and business with multiple stakeholders, both public and private; paradigms rewarding the bioentrepreneurship of biotech companies; service industries from the human genome sciences to multinational agribusiness complexes; corporate models from the ground up, including accountability practices and the obsession with quantification; the legal-juridical complex; and the insurance industry (Murray, et al., 2007). This list is by no means exhaustive, but it indicates that the challenges are legion. Nothing less than a multitude of micro-resistances is called for in each of these domains. In the face of a strategic fundamentalism that closes off debate, we must be mindful to resist in such a way that we open up critical debate and question those mechanisms that work to seduce us into complacency. In short, the health care sciences ought to work to foster an ethic of patient care that resists technocracy, that is, an ethic that will be respectful of and responsible for patient diversity for the good life (Murray, 2007).
In other words, like all good “microfascism” apparently EBM requires absolute “conformity” and enforces it by ruthless means. This is such utter nonsense on so many levels that it’s as breathtaking in its hyperbole as the original “microfascism” article while at the same time being yawn-inducing in its utter tedium. Yes, yes, we know. That nasty science and that nasty EBM construct restrict your freedom to consider any source of evidence you want, no matter how dubious. All this entire attack is is the “health freedom” movement dressed up in incomprehensible “diversity”-promoting postmodernist jargon to an even more laughable extreme than their previous articles.
Running as a thread throughout the entire article, I could also discern the “science was wrong before” argument so beloved of cranks of all stripes, along with a lot of other logical fallacies. Once again, what matters more than anything else are the political and social implications, because EBM must obviously controlled by all the things the they hate, the insurance industry, “bioentrepreneurship” and big pharma, the veritable Roots of All Evil and Threats to Diversity. To achieve their aim of destroying EBM (with a minor concession that in their ideal world there would “still be a role” for RCTs, although their clear implication is that that role will be small), the whole argument boils down to a predictable appeal to other ways of knowing, in the form of advocating “resisting technocracy” and, above all else, promoting “diversity” (whatever that means). In reality, “diversity” appears to mean nothing more than the freedom of practitioners to believe or do whatever they want, science and evidence be damned–because, as people like Murray et al tell us, “evidence” is hopelessly corrupted by the power structure, whatever that means. Science is therefore nothing more than just one other way of knowing, with no inherent superiority to any other way of knowing how nature works or, on a practical level, what treatments work the best. Although Murray et al don’t come right out and say it, the implication of their argument is that if a quack believes that liver flukes cause cancer and that zapping the flukes with a cheap electrical device will cure all cancer, who is science or that nasty EBM to say that that’s not true? In any case, two years later, I still stand in awe of the amount of sheer antiscientific nonsense concentrated into such a brief little paper, and today I stand even more in awe that Murry, Holmes, and Rail have matched it.
“A more fluid concept of evidence”
You may laugh at the shenanigans embodied in articles like the two from which I have just liberally quoted, and indeed they are laughable in a great many ways, with their ludicrously dense, overblown, jargon-laden, pretentious rhetoric. But underneath all that jargon lie arguments very similar to the ones being used by advocates of unscientific medical practices and, yes, outright quackery, many of whom are opposed to science-based medicine in practice, although they will never admit it. For example, let’s consider the case of David L. Katz, MD, MPH, FACPM, of FACP, is founder and director of the Integrative Medicine Center (IMC) at Griffin Hospital in Derby, Connecticut, who is also an Associate Professor, Adjunct, of Public Health and director of the Prevention Research Center (PRC) at the Yale University School of Medicine. My co-blogger Dr. Kimball Atwood IV certainly has. Here’s the statement that Dr. Katz has become infamous for:
I think we have to look beyond the results of RCTs in order to address patient needs today, and to do that I’ve arrived at the concept of a more fluid form of evidence than many of us have imbibed from our medical educations…[Referring to a patient anecdote, Dr. Katz went on.] Now, we don’t want you on narcotics anymore than you want to be on narcotics. We initicated a course of acupuncture and over the next two to three months weaned him off narcotics. He was pain-free on acupuncture and subsequently transitioned into homeopathy. Now, I don’t care to get into a discussion of how or even whether homeopathy even works, but this guy had tried everything.
[...]
And the anecdotal evidence that homeopathy might be effective was brought up by the naturopaths. We tried it. It worked.
Of course, Dr. Katz neglects to describe exactly how he knew it “worked,” namely anecdotal evidence which, as we have seen many times in this blog, is incredibly prone to misinterpretation and confusing correlation with causation, thanks to a number of cognitive quirks we humans all share. Indeed, science itself, not just science-based medicine is designed to minimize the effects of these quirks. In response to criticism, Dr. Katz went even further:
The view I expressed, and that guides our practice, is that human need goes on long after the results of randomized clinical trials start to run thin. I do not think doctor and patient should part company there. I believe that responsible use of the science we have should not preclude responsiveness to the needs of patients that fail to respond as textbooks say they should to that all-too-limited science. I believe, in other words, that patient need, not trial results, should be the ultimate master medical care must serve.
I learned devotion to evidence-based practice from my teachers and professional colleagues; I was pushed toward integrative medicine by the needs of my patients.
See what I mean? Just like the postmodernist, Dr. Katz views science and EBM as nothing more than just another “narrative” that he can abandon when he sees fit or, as he puts it, when he perceives patient need to drive him to do so. Like postmodernists, he also views words as being malleable with meanings that can be “deconstructed” (in practice, changed) at will. Indeed, as Dr. Atwood correctly points out, these are subtle distortions of language that are very powerful and incredibly useful for CAM advocates. RCTs, “what we know,” and science are all conflated as though they mean the same thing, leading to a false dichotomy in which the only choice when the edges of what RCTs tell us are reached is to rely on anecdotes, testimonials, and other forms of unreliable and subjective “evidence” in order to justify choosing unscientific treatment modalities, even abject pseudoscience like homeopathy. Indeed, Dr. Atwood, as usual, put it well when he wrote:
It also follows that in the much larger realm of practices outside “that all-too-limited science,” there is no basis for distinguishing one from another. Since science has nothing to say about them, all are equally promising. All are thus fair game for the practitioner “pushed toward integrative medicine by the needs of [his] patients.”
Postmodernist, indeed. When you come right down to it,in practice there is no real difference between the postmodernist critique that EBM or scientific medicine represents nothing more than one method or “narrative” among many, all of which are equally valid, depending upon your point of view, and Dr. Katz’s apparent view that scientific medicine is simply another tool in his armamentarium, to be abandoned when he sees fit, all in the “name of the patient,” of course. Katz’s excuses have their origin in the very same sorts of fallacies that postmodernism leads to when applied to science or the scientific method, and, although it doesn’t sound as obviously ridiculous as the postmodernist articles I quoted, it is.
Similarly, the postmodernist criticism of EBM as nothing more than a “procrustean” methodology that ruthlessly enforces “conformity” and crushes the autonomy of the individual practitioner echoes the laments of CAM advocates who do not like scientific medicine because it fails to support the brand of woo they like. Indeed, all manner of advocates of antiscience in medicine invoke the same refrain: homeopaths, reiki practioners, “distance healers,” and many others. And let’s not forget HIV/AIDS denialists, who insist that the “dogma” that HIV causes AIDS is not based on science but rather ideology–and, of course, a big pharma conspiracy to make tons of money selling expensive antiretroviral cocktails or an academic conspiracy to bring in tons of grant money (or both)–and is ruthlessly enforced, crushing any who would “question it,” such as Peter Duesberg. Some HIV/AIDS denialists have even likened the establishment a “criminal enterprise” in “crushing dissent.” Indeed, antivaccinationists use similar rhetoric labeling vaccines a big pharma conspiracy to make lots of money and routinely argue that science is wrong and anecdotes should trump science, most infamously when Jenny McCarthy exclaimed that her son Evan “is my science.” What is such an argument other than that personal experience telling “Dr.” McCarthy that vaccines caused her son’s autism is a “narrative” that’s just as valid as the “narrative” of the scientific evidence that says they almost certainly did not?
It’s not just medicine, either. Pseudoscientists and antiscientists such as creationists invoke variations of postmodernist criticism too. In all fairness, I have been told by some who have studied postmodernism that these invocations are almost always bad postmodernism, but they are used nonetheless. Generalizing further, it’s not even just medicine or science; it can be history too. Think Holocaust deniers, JFK assassination conspiracy theorists, or even 9/11 Truthers. On favorite variant of this gambit is to label science as “dogma” or “religion,” particularly a “fundamentalist” religion. Of course, one key difference between science and fundamentalist religion is that science is inherently self-correcting. True, it may take a long time and be a very messy process, particularly in medical science, but eventually in medicine treatments found through science and clinical trials to be ineffective or to have too many side effects for their benefit are discarded. Contrast that to “alternative” medicine, where prescientific ideas about how the body works (vitalism, for instance) and religious ideas (what are reiki and other forms of “energy healing” but a form of faith healing, in which the healer is viewed to have some skill at manipulating “life energy” for therapeutic effect?) withstand all evidence showing them to have no basis in science or efficacy above that of an elaborate placebo. Of course, that’s why postmodernism is so attractive to CAM aficionados. If these prescientific ideas are just another narrative with no objective means of showing it to be a less valid description of reality than medical science, there is no reason to prefer science over such woo. In such a relativistic worldview, then it’s easy to make the appeal to us “rigid” and “close-minded” scientists to be “open-minded” to the point that our brains fall out.
For CAM practitioners and all advocates of pseudoscience and antiscience, whether they realize it or not, postmodernist assaults on the very basis of science are the gift that keeps on giving. After all, to the CAM advocate, it’s all about changing the rules of evidence when analysis under the current rules show your favored therapy to be nonsense.

Dichotomous thinking and uncertainty in medicine and science
Medicine is by its very nature uncertain. Unfortunately, humans don’t deal well with uncertainty, and our tendency towards dichotomous thinking leads us to think that if we’re not absolutely certain about something we don’t know anything.
David Gorski on November 11, 2019
Link

If there’s one thing that I’ve learned about communicating science and medicine, in particular countering pseudoscience and quackery, it’s that belief in pseudoscience is very difficult to shake. Anyone who tries to counter, for example, antivaccine misinformation will soon find that simply countering misinformation with good information doesn’t work, at least not with the hard core antivaxers. The same is true when trying to counter cancer quackery and many other forms of medical pseudoscience. Certainly one thing that contributes to this resistance is a very human trait to crave certainty and become anxious when there’s uncertainty. Last week, I came across an article by a psychotherapist named Jeremy Shapiro—he’s at my old stomping grounds of Case Western Reserve University, too!—that delved into this question and led me to decide to write about it for this week. It’s a tendency that is at the root of a lot of science denial, not just pseudoscience and quackery in medicine but denial of climate science, evolution, and many other forms of science denial, which he attributes to the same basic thinking error.
Before I get into the article itself, I tend to like to refer to this particular thinking error as: If we don’t know everything, we know nothing! In fact, I gave an example of this sort of thinking just last week but didn’t really discuss it. Basically, Bill Maher, when interviewing Dr. Jay Gordon, kept harping on the uncertainty in medicine as a reason to doubt the safety and efficacy of vaccines, to doubt the conclusions of science, listing examples of dietary recommendations that changed, of Accutane being withdrawn from the market, and various other examples of when new findings led to significant changes in medical practice. Basically, his entire line of “reasoning” if you can call it that, was exactly the flaw in thinking that I listed above: If we don’t know everything then we know nothing, but with an additional point. If we don’t know everything about everything in medicine, then anything is possible, no matter how much evidence against it exists; e.g., a link between vaccines and autism. At the end of the interview segment, he even explicitly said that, unless a doctor can tell him exactly what causes cancer and exactly how to cure it then he won’t shut up about asking questions about medical issues. Basically, Maher kept ranting about what we don’t know about medicine, completely ignoring how much we do know.
Shapiro characterizes this sort of thinking a bit differently. After first noting that science deniers do cite science and empirical evidence but cite it in invalid and misleading ways, he notes that dichotomous thinking, also referred to as black-and-white or all-or-none thinking, is a characteristic factor in a number of mental conditions, including depression, anxiety, aggression and borderline personality disorder. This type of thinking involves taking a spectrum of possibilities and dividing it into two possibilities, eliminating shades of gray. Everything is either black or white.
Then:
Spectrums are sometimes split in very asymmetric ways, with one-half of the binary much larger than the other. For example, perfectionists categorize their work as either perfect or unsatisfactory; good and very good outcomes are lumped together with poor ones in the unsatisfactory category. In borderline personality disorder, relationship partners are perceived as either all good or all bad, so one hurtful behavior catapults the partner from the good to the bad category. It’s like a pass/fail grading system in which 100 percent correct earns a P and everything else gets an F.
In my observations, I see science deniers engage in dichotomous thinking about truth claims. In evaluating the evidence for a hypothesis or theory, they divide the spectrum of possibilities into two unequal parts: perfect certainty and inconclusive controversy. Any bit of data that does not support a theory is misunderstood to mean that the formulation is fundamentally in doubt, regardless of the amount of supportive evidence.
Similarly, deniers perceive the spectrum of scientific agreement as divided into two unequal parts: perfect consensus and no consensus at all. Any departure from 100 percent agreement is categorized as a lack of agreement, which is misinterpreted as indicating fundamental controversy in the field.
This is exactly the way antivaxers appear to think. Those of you who regularly encounter antivaccine misinformation will be able to discern this pattern in the arguments made. If a vaccine isn’t absolutely 100% safe, it’s dangerous, toxin-laden crap. If a vaccine is not 100% effective at preventing the disease it’s designed to prevent, it’s utterly useless. Any vaccine failure at all is evidence to them that they are correct, which is why they constantly harp on outbreaks in which vaccinated children fall ill as “evidence” that vaccines are useless, often crowing about how more vaccinated children became ill than unvaccinated. Of course, when they do this, they completely ignore the inconvenient fact that there are many more vaccinated children than unvaccinated children, which means that when you look at the percentage of unvaccinated children who fall ill compared to the percentage of vaccinated children who fall ill you’ll always find that unvaccinated children are far more likely to fall ill than vaccinated children. It’s a gambit that persuades because many people aren’t that great at math and won’t think in fractions, percentages, and probabilities of becoming ill without prompting and having someone lead them through the calculation.
Dichotomous thinking also has a consequence in how one evaluates existing scientific evidence:
Proof exists in mathematics and logic but not in science. Research builds knowledge in progressive increments. As empirical evidence accumulates, there are more and more accurate approximations of ultimate truth but no final end point to the process. Deniers exploit the distinction between proof and compelling evidence by categorizing empirically well-supported ideas as “unproven.” Such statements are technically correct but extremely misleading, because there are no proven ideas in science, and evidence-based ideas are the best guides for action we have.
I have observed deniers use a three-step strategy to mislead the scientifically unsophisticated. First, they cite areas of uncertainty or controversy, no matter how minor, within the body of research that invalidates their desired course of action. Second, they categorize the overall scientific status of that body of research as uncertain and controversial. Finally, deniers advocate proceeding as if the research did not exist.
This is one reason why I almost never use the word “proof” when discussing science, even when discussing homeopathy which has been about as close to proven to be impossible as anything can be in science. I discuss evidence, not proof. In any event, let’s apply this example to antivaxers. Again, like the example above, where I pointed out that if a vaccine is not 100% effective and safe it’s dangerous and ineffective (to them), if there is any controversy in the science surrounding a vaccine, no matter how minor, then the “science isn’t settled”. (How many times have you heard that line from antivaxers—or, come to think of it, quacks and other science deniers?) The “controversy” doesn’t even have to be a legitimate one, either. Take the example of whether vaccines cause autism. All the large, well-designed, well-executed epidemiological studies have failed to find a whisper of a whiff of a hint of a signal of a correlation between vaccination and autism. However, there do exist studies that have found a correlation. The problem is that they’re all studies by antivaxers, such as Andrew Wakefield, Mark and David Geier, and others, and they’re all terrible studies with huge methodological flaws. As discussed by Shapiro, though, antivaxers give equal (or even greater) weight to the studies by antivaxers finding that vaccines cause autism as they do to the vast compendium of studies by legitimate scientists that find vaccines don’t cause autism, and conclude that there’s still a scientific controversy over whether vaccines cause autism. There isn’t.
There’s a similar pattern with cancer quackery. Because I specialize in the surgical treatment of breast cancer, I understand that it’s incredibly scary to be diagnosed with cancer, even a cancer like breast cancer, which most of the time can be treated with a high likelihood of long-term survival or “cure,” if you will. Most patients, even those who believe in cancer quackery, will accept that surgery “works”, because it’s fairly intuitive that removing a cancer will treat it, but far more have a problem with chemotherapy and radiation. That’s why there are so many women with alternative medicine cancer cure testimonials who, when you examine their stories more closely, turn out to have accepted surgery but refused chemotherapy and/or radiation and were, in essence, lucky enough to have been “cured” by the surgery. In any event, the same sorts of arguments are common. If there isn’t a 100% cure rate, then conventional cancer treatment is useless and a dangerous mix of “cut, burn, poison”. Meanwhile, cancer quacks promise a 100% cure rate or a 90% cure rate for cancers that conventional medicine can’t cure but only manage, and any study that a treatment has more toxicity or is less effective than previously suspected is touted as evidence that chemotherapy doesn’t work. (It does.)
Shapiro points out the same phenomenon in another area of science. See if you hear echoes of arguments we’ve dissected on this blog before:
This same type of thinking can be seen among creationists. They seem to misinterpret any limitation or flux in evolutionary theory to mean that the validity of this body of research is fundamentally in doubt. For example, the biologist James Shapiro (no relation) discovered a cellular mechanism of genomic change that Darwin did not know about. Shapiro views his research as adding to evolutionary theory, not upending it. Nonetheless, his discovery and others like it, refracted through the lens of dichotomous thinking, result in articles with titles like, “Scientists Confirm: Darwinism Is Broken” by Paul Nelson and David Klinghoffer of the Discovery Institute, which promotes the theory of “intelligent design.” Shapiro insists that his research provides no support for intelligent design, but proponents of this pseudoscience repeatedly cite his work as if it does.
Again, this is part of how this sort of thinking works. The core tenets of the theory of evolution are supported by an enormous body of mutually-reinforcing evidence from a number of different disciplines built up over many decades. The controversies in evolution, such as they are, tend to be at the bleeding edge of the science, and the bleeding edge is always way more uncertain than the core. (Otherwise they wouldn’t be bleeding edge.) Yet creationists use those scientific controversies to cast doubt on the very core of evolution. It’s how science denial works. Similarly, climate science deniers use controversies at the very edge climate science to cast doubt on the entire conclusion of climate science that the earth’s climate is warming catastrophically largely due to human activity. Like the case with antivaxers, too, often the deniers produce dubious scientific studies to give the illusion of scientific controversy.
As Shapiro concludes:
There is a vast gulf between perfect knowledge and total ignorance, and we live most of our lives in this gulf. Informed decision-making in the real world can never be perfectly informed, but responding to the inevitable uncertainties by ignoring the best available evidence is no substitute for the imperfect approach to knowledge called science.
Indeed.
My one quibble with Shapiro is that I’m not so sure that a pathological (or near-pathological) level of dichotomous thinking is necessary for science denial to take hold, just a normal level coupled with perhaps an above-average need for certainty. When discussing this aspect of science denial, I like to quote a song by David Bowie, Law (Earthlings on Fire): “I don’t want knowledge. I want certainty!” That pretty much sums it up. If there’s a trait among humans that strikes me as being universal, it’s an unquenchable thirst for certainty. It’s a major force that drives people into the arms of religion, even radical religions that have clearly irrational views, and it isn’t expressed only through extreme religiosity. As anyone who accepts science as the basis of medical therapy knows, there’s a lot of the same psychology going on in medicine as well. This should come as no surprise to those committed to science-based medicine because there is a profound conflict between our human desire for certainty and the uncertainty that is always inherent in so much of our medical knowledge. The reason is that the conclusions of science are always provisional, and those of science-based medicine arguably even more so than many other branches of science. Why? Because medicine involves applying imperfect science to the treatment of disease. Often that application produces clear-cut cures. Arguably more often, though, the results are more mixed and less satisfyingly clear-cut (e.g., the treatment of chronic diseases). To go back to the David Bowie quote, evidence is knowledge, not certainty or, to echo Shapiro, “proof”. Take that craving for certainty and mix in some dichotomous thinking and any conclusion of medicine that isn’t 100% certain becomes very uncertain.
As I’ve said before, one of the hardest things for the average person who is not medically or scientifically trained to accept about science-based medicine is that the conclusions of science are always subject to change based on new evidence, sometimes so much so that even those of us “in the biz” can become a bit disconcerted at the rate at which knowledge we had thought to be fairly settled changes. One example that I frequently like to cite is how duodenal peptic ulcer disease (PUD) was treated 35 years ago compared to how it is treated now. Between 1984 and 1994, a revolution occurred on the basis of the discovery of H. pylori as the cause of most of the gastric and peptic ulcer disease we see. Where in 1985 we treated PUD with H2-blockers and other drugs designed to block stomach acid secretion, now antibiotics represent the mainstay of treatment and are curative at a much higher success rate than any treatment other than surgery and without the complications of surgery. I’m sure any other physician here could come up with multiple other examples. In my own field of breast cancer surgery, from time to time I look back at how we treated breast cancer nearly 30 years ago, when I first started residency, and compare it to how we treat it now, and I marvel at the changes, many of which I had to learn after having completed my training. If such changes can be disconcerting even to physicians dedicated to science-based medicine, imagine how much more disconcerting they are to lay people, particularly when they hear news reports of one study that produces one result, followed just months later by a report of a different study that gives a completely different result. That’s definitely not certainty!
The problem is that quacks offer what humans crave: Certainty. They also offer it in a dichotomous manner: My quackery is good and effective; conventional medicine is a useless and toxic (and big pharma profits). Unfortunately, as I’ve discussed before, scientists often fall prey to what has been called the “truth wins” assumption. This assumption, stated simply, is that when the truth is correctly stated it will be universally recognized. Those of us who make it one of our major activities to combat pseudoscience know, of course, that the truth doesn’t always win. Quite the contrary, actually, I’m not even sure the “truth” even wins a majority of the time — or even close to a majority of the time. Moreover, most recommendations of science-based medicine are not “truth” per se; they are simply the best recommendations physicians can currently make based on current scientific evidence. They have changed. They’re changing now. They will continue to change. The examples are endless: mammography recommendations, treatment for hypercholesterolemia, adjuvant chemotherapy recommendations for breast cancer and other cancers. Unfortunately, there are quite a few doctors who are just as uncomfortable with change as the average person and still use out-of-date treatments and techniques.
The challenge, then, as a physician and science communicator, is twofold. First, we have to be comfortable when dealing with uncertainty and change ourselves. If we can’t, then there’s no way we’ll be able to communicate uncertainty. Second, we have to be careful to acknowledge and explain the uncertainty in the findings of science, noting where there is little to no uncertainty and where there is more. Of course, by the time we are adults, it’s often too late to get that message across in a way that we ever really internalize it. We really need to teach our children not just critical thinking but that in science there is no such thing as absolute proof and that medical and scientific conclusions are supported by evidence and subject to change and revision in the face of new evidence. Then there would be less for quacks to work with when they try to persuade people that science is unreliable and their treatments provide certainty.

The Human Mold: Another Example of Self-Deception
Harriet Hall on September 16, 2014
Tweet


José Jarimba believes that our bodies are physically molded into an asymmetric form by our mothers’ sleeping positions during pregnancy, that this has lifelong adverse impacts on health, and that shoe inserts can eliminate pain and other health problems by realigning the body. This is a silly untested hypothesis by a single individual. As such, it would be too minor to merit mention on SBM; but it is worth analyzing as a teaching opportunity. Jarimba attempts to bypass the scientific process; he provides a prime example of self-deception, confirmation bias, scientific ignorance, and the “Unpersuadables” I recently wrote about.
Much of alternative medicine originated with a “lone genius” who had an epiphany, thought he had discovered something no one had ever noticed before, extrapolated from a single observation to construct an elaborate theory that promised to explain all or most human ills, and began treating patients without any attempt to test his hypotheses using the scientific method. Some of them were uneducated laymen, others were scientifically trained medical doctors who should have known better. I wrote about one of them here, Dr. Batmanghelid, inventor of the Water Cure, who attributed a great variety of illnesses to dehydration after he thought he had cured a prisoner’s peptic ulcer disease by giving him a glass of water. Similar paths were followed by many others. Hahnemann invented homeopathy after he thought a malaria remedy gave him symptoms of malaria. Palmer invented chiropractic after he thought he had restored a man’s hearing by repositioning an out-of-place bone in his back. Nogier invented ear acupuncture after he imagined that the external ear looked sort of like a fetus. Shapiro invented EMDR after she noticed during a walk in a park that moving her eyes seemed to reduce the stress of disturbing memories. Bach invented Bach flower remedies after a walk in the country revealed his intuitive psychic connection to various plants. Jose Jarimba follows in their footsteps.
Jarimba has no medical or scientific training. His epiphany was a combination of two observations: he noticed that a man’s sideburns were asymmetrical and his hair coarser on one side, and he watched a film clip about a fetus. He thought: adults are not symmetrical, and when a mother lies on one side, that puts asymmetrical pressure on the fetus’ body. Eureka! He concluded that the mother’s sleeping position must cause physical deformation, organ deformation and malfunction, and various ailments.
He asked, what do pregnant animals do in the wild? He says that mammals alternate sleeping on one side and the other (Do they? How does he know? Are other mammals more symmetrical than humans?) and birds sleep on their feet because if they lay down their eggs would come out lopsided. (I think it’s because if they tried to lie down on their side they would fall off the branch where they are roosting.) He says pregnant women should sleep equal numbers of hours on the left and right side if they want a healthy baby.
He thinks the process of deformation begins at conception. He blames modern technology. Before we had machines, women worked and exercised. He thinks their fetuses were in an upright position and faced towards the mother. (Really? I don’t think so!) After the development of machines, pregnant women were advised to rest and got help with their domestic duties. Those who worked and were active throughout pregnancy had more symmetrically perfect babies. (Really? How does he know?)
He claims that physical misalignment occurs in proportion to the hours the mother sleeps on one side. 6 hours a night corresponds to 5-10% chance of misalignment, 16 hours to 60-70%. (Who sleeps 16 hours?) I asked him how he figured out those percentages. He explained that a day has 24 hours, and if a mother rests 8 hours a day, that’s 33 1/3%; then divide one inch by 3, and 8 hours equals one third of an inch. I asked where the figure of one inch came from, and he said that’s the measurement of deformity that can be corrected without surgery, and he cited a webpage about congenital hip dysplasia. (None of this makes a bit of sense to me; maybe it will to you.)
If the mother sleeps on her right side, he says it affects the baby’s left side. He seems to be assuming the baby’s left side will be down. The most common presentation at delivery is left occiput anterior, which would put the baby’s left side down when the mother slept on her left side. And the baby’s position is constantly changing. It only becomes fixed when the presenting part engages in the pelvis shortly before delivery; most breech babies turn head-first spontaneously by the 37th week. He thinks the internal organs are affected. He thinks the valves inside the heart can be malformed by fetal position; he thinks they become tilted towards the affected side and that this decreases oxygenation and cause heart attacks. He says resting on one side can cause upper body misalignment and on the other side, lower body misalignment (I asked him to explain how that would work, and he failed to answer.) He says sitting on a chair causes foot misalignment (how would that work if the baby was in the common head-down position?). He says vision is affected by position because the orbital muscles on one side become stretched toward the back of the head. He thinks too much sitting leads to breech deliveries. None of this is plausible; it’s impossible to visualize how gravity and pressure could accomplish those things.
I had never heard of Jarimba before he contacted me and sent me a copy of his book The Human Mold. We had a long e-mail exchange. I asked him what left-sided asymmetry meant and how he could determine which side was the abnormal one. His answer was nonresponsive: “When I say asymmetric from the left side, I mean that the left side is the abnormal side.”
I asked how he knew how many hours a woman slept on one side. He acknowledged that self-reports were unreliable, but claimed he could assess sleeping time accurately by measuring the degree of physical misalignment. He can tell the degree of misalignment from the number of hours, and he can tell the number of hours from the degree of misalignment. I pointed out that this is circular reasoning. He did not comment.
He says most women sleep on their right side “as recommended by the CDC.” He got that backwards: the left side is preferable, although not enough to matter. We know that in late pregnancy, the uterus may press on the vena cava, and lying on the back or on the right side may reduce blood flow; but in a normal pregnancy this is not significant. Women are encouraged to sleep in whatever position they find most comfortable.
He described an experiment he did with 17 women, asking them to alternate sleeping sides and finding that the babies were born more symmetrical. There was no blinding and no control group. When I asked him about that, he said, “I do understand very well that experiments with no control group might reach false conclusions, but in this case, I am proud to announce that my research produced no false conclusions.”
I asked how asymmetry could affect organ function, and he said when the structure is misaligned the weight of the imbalanced fluid pressing on the veins causes the veins to bend resulting in slow blood flow to the organs, heart, head (brain), and to all the body causing major health complications. He says a “tilted” kidney can’t function properly. I don’t believe any of those assertions.
I asked if he had talked to any obstetricians about his theories, and he said no, because he had studied their literature and found “a lack of deep knowledge into the development of our being.” His own lack of even basic knowledge is obvious in the cover illustration he chose to grace his book. It depicts a fetus in a woman’s abdomen. The fetus’ head is positioned just under the breasts at a height not reached even at full term. The fetus appears to be at about the 12 week stage of development and at that stage the uterus should be about the size of a grapefruit and still down inside the pelvis, and the fetus should be much, much smaller in relation to the woman. And the fetus’ position as shown is that of a footling breech, one of the rarest presentations for delivery.
He has treated over 2,000 clients. He doesn’t advertise, but he has an office where he sees them by appointment only. He charges $120, sees them only once, and gives them a shoe insert that he makes himself. He does no systematic follow-up.
He says you can analyze yourself by looking in a mirror, and you can analyze your friends. You will notice that one ear sticks out more than the other, one shoulder is higher than the other, one eye is smaller than the other (he says this is more noticeable at the end of the day), one leg is shorter than the other, one heel is more curved, etc. Once you know which is your bad side, you can sleep on the other side to counteract the asymmetry. He says the blood flows slower on your bad side.
He himself had 52 years of low back pain, foot pain, and overall body discomfort. All of these vanished after he put an insert in his shoe. He has plenty of patient testimonials. Back pain relieved. Back surgery avoided. More energy. Better sleep, more mentally focused, no headaches, no longer get sick. Alternating sleep sides during pregnancy produced a happy baby who doesn’t spit up and sleeps through the night. He apparently doesn’t understand that the plural of “anecdote” is not “evidence.”
He echoes the claims of chiropractors about spinal alignment, nerve interference, and disease. He cites the discredited Winsor Autopsies study from 1921 that supposedly correlated spinal misalignments with diseases of internal organs and concluded that 80% of chronic disease problems are linked to the spine. Jarimba comments that Winsor’s findings “indicate that most human physical deformities found in his autopsies were possibly caused by the mothers of the subjects sleeping/resting mainly on one side during pregnancy.” Winsor’s findings indicate no such thing!
He indulges in a bit of doctor-bashing, encouraging people to avoid dependence on the current health system. He says that reading media reports and visiting a doctor can cause stress and “lower” the immune system. He says overuse of medical care is wrecking your health. He says the mere act of purchasing health insurance may subconsciously prepare the mind and body to allow your health to break down. He says maintaining a positive attitude will help keep you healthy.
I suggested he might be risking prosecution for practicing medicine without a license. He said, “I do not practice medicine. I do not touch people physically.” Yet he is telling prospective clients he can relieve their pain and emotional symptoms, and on this Facebook page his occupation is listed as Family Doctor.
I asked him a lot more questions, but instead of answering, he replied, “I am sorry, since you haven’t witnessed my work in action (live), I see no need to keep on going.”
I eventually got him to answer 3 more questions:
     1.  “Has what I wrote made you question your theory in any way?” “No.”
     2.  “Do you think there is any possibility that you might be wrong?” “No, because it works, and if it works it is OK.”
     3.  “If good scientific studies found that time sleeping on one side was not related to asymmetry, and that your shoe inserts worked no better than placebo, would that change your mind?” “There is not a single possibility of that happening.”
Conclusion
In his book, Jarimba says, “I hope that my observations will one day have a positive impact in the health system and the entire human population.” I think there is little chance of that.
This is yet another sad example of a person who has deceived himself. He has adopted a belief without ever putting it to a proper test, and has been fooled by his own experiences and by confirmation bias. He has a poor understanding of science, of critical thinking, and of the cognitive errors we humans are all subject to. He fails to understand what Druin Burch calls “Medicine’s beautiful idea,” the idea that we can easily be misled into false beliefs and must test even the most apparently obvious hypotheses using reliable scientific methods. Jarimba demonstrates the hubris of a layman who thinks he knows more than the experts, the Dunning-Kruger effect. He demonstrates the closed mind of an “unpersuadable” to perfection. Fortunately, he is only a single crank who has not yet spawned a movement, and what he does is not likely to harm his clients. He may even do some good here on SBM by serving as a bad example.

Don’t Believe What You Think
A new book by Edzard Ernst provides a concise course in critical thinking as well as a wealth of good science-based information to counter the widespread misinformation about SCAM.
Harriet Hall on May 19, 2020
Link

Edzard Ernst was once a practitioner of alternative medicine. As the world’s first professor of complementary medicine (at the University of Exeter, UK) he led a research team to scientifically investigate what worked and what didn’t and discovered that there was little or no evidence behind most of the claims of alternative medicine. Today he is arguably the world’s foremost expert on alternative medicine and has become its harshest and most prolific critic. I reviewed his autobiography A Scientist in Wonderland: A Memoir of Searching for Truth and Finding Trouble which tells the story of his transformation and the opposition he encountered.
His output of books on so-called complementary and alternative medicine (SCAM) is impressive. I have reviewed several of them:
    • Trick or Treatment: The Undeniable Facts about Alternative Medicine, which he co-authored with Simon Singh,
    • Alternative Medicine: A Critical Assessment of 150 Modalities
    • Homeopathy: The Undiluted Facts
    • SCAM: So-Called Alternative Medicine
    • More Harm Than Good: The Moral Maze of Alternative Medicine, which he co-authored with a medical ethicist.
His newest book is Don’t Believe What You Think: Arguments for and Against SCAM. It is, as he describes it, a crash course in critical thinking. And much more.
Why do people turn to alternative medicine?
Ernst explains,
Enthusiasts of SCAM tend to hope for fewer side effects, symptom relief, a cure for their condition, improvements in the quality of life, and protection from illness. Such high expectations are usually based on misinformation, often even on outright lies.
In this book, Ernst shows how consumers get manipulated into believing things that are evidently wrong. Those who advocate false beliefs, whether about climate change, evolution, or vaccines, rely on the same tactics:
    • They ignore the scientific consensus.
    • They cherry-pick their evidence.
    • They rely on poor quality studies, opinion, and anecdotes.
    • They invent conspiracy theories.
    • They defame their opponents.
    • They point out that science has been wrong before.
    • They say, “science doesn’t know everything.”
He concisely and eloquently covers the psychology behind bias:
    • Information overload: consumers are bombarded with untruths.
    • Motivated ignorance: people choose to avoid disconcerting information.
    • Motivated reasoning: finding reasons to defend a treasured belief regardless of evidence against it.
    • Confirmation bias. This includes phenomena like anchoring, the bandwagon effect, Dunning-Kruger, illusory correlation, subjective validation, post-purchase rationalization, and others.
    • Denialism: when confirmation bias degenerates into a complete denial of the truth.
After a general discussion of critical thinking, he gets into particulars. In each of 35 essays he addresses a commonly held belief or argument about SCAM and demolishes it with reasoning and scientific evidence. Here’s a sampling:
“I have done my research”
How often have we heard this? But research doesn’t mean what they seem to think it means. What they have done is more properly called “exploration.” Research should be systematic. This “research” was not systematic: they looked at randomly chosen websites, articles, videos, and/or books, and then uncritically accepted the views that appealed to them. After their exploration, some of them favor an “integrative” approach, thinking that will achieve the best of both worlds. Ernst provides the text of an announcement about Integrative Cancer Therapy (ICT) from a German academy claiming that ICT is holistic, addresses the patient’s personal needs, provides individualized treatments, and other claims. He numbers the claims 1 through 10, and comments on each of the claims by number, based on reliable research. His comments:
     1.  Actually, this describes conventional oncology!
     2.  Actually, this describes conventional oncology!
     3.  Actually, this describes conventional oncology!
     4.  Actually, this describes conventional oncology!
     5.  Actually, this describes conventional oncology!
     6.  Actually, this describes conventional oncology!
     7.  Actually, this describes conventional oncology!
     8.  Actually, this describes conventional oncology!
     9.  Actually, this describes conventional oncology!
     10.  Actually, this describes conventional oncology!
SCAM frequently tries to co-opt conventional treatments such as exercise and pretends they are discoveries of SCAM.
SCAM is natural
All SCAMS claim to be natural, but are they really? Thinking critically makes one question what is natural about ear candles, homeopathy’s Berlin Wall remedy, naturopathy’s neural therapy (injection of a local anesthetic), chiropractic’s moving joints outside their natural range of motion, etc.
SCAM can save us all a lot of money
The arguments supporting this claim are all fallacious. Most of the published studies (and there are a lot of these) show that SCAM use causes extra expense.
SCAM providers offer sound advice
There is plenty of evidence that much of their advice is not just poor, but also dangerous, and sometimes even life-threatening. Ernst encourages readers to search for their own examples, which are easy to find.
This study proves…
Sometimes a study finds bizarre results that can’t be attributed to any obvious flaws. It may have been replicated, but it may not have been an independent replication but have been done by the same original authors. There are four possibilities: fraud, coincidence, undetected bias, or the study is correct and the previous consensus was wrong. The last possibility is the least likely: one anomalous study can’t overturn decades of previous research.
SCAM practitioners don’t diagnose disease
They may say they don’t diagnose, but in reality they use a variety of bogus diagnostic methods including dowsing, applied kinesiology, iridology, Kirlian photography, live blood analysis, pulse diagnosis, tongue diagnosis, and the Vega test and similar variants of electroacupuncture according to Voll (EAV). These have all been tested and have been thoroughly debunked.
Imaginary dialogs
SCAM proponents will find reasons to deny the evidence and will argue about everything in this book. Ernst has a wealth of experience countering their arguments and has unique insight into their thinking. He presents imaginary dialogs between a scientist and a proponent of SCAM that are very true to life and illuminate the conflict.
Conclusion: Don’t believe what you think
        ◦ This is a very useful, insightful, well-written book that is packed with reliable, science-based information. If you have believed any of the misinformation about SCAM, this book might change your mind. If you are not a believer, it may give you good tools to help you explain to believers that what they think they know is based on misinformation. Whatever you think about SCAM, don’t believe what you think. Think again and start doubting your belief. As Feynman said, “The first principle is that you must not fool yourself and you are the easiest person to fool.”

The Unpersuadables
Harriet Hall on September 2, 2014
Link

We would like to believe people are rational. We would like to believe that if they have formed a false belief based on inaccurate information and poor reasoning, they will change that belief when they are provided with accurate information and better reasoning. We are frequently disappointed.
An example of what should happen
I recently talked with a college professor who believed chiropractic treatment could lower blood pressure. His belief was based on a media report of a chiropractic study. He thought it was plausible that neck manipulation could somehow relieve obstructions to blood flow to the base of the brain, thereby somehow correcting the cause of high blood pressure. I told him that rationale was anatomically and physiologically implausible. I pointed out that the researchers used NUCCA, a form of manipulation that is rejected by most chiropractors. He did not know what NUCCA was. I provided him with information, including links to the study itself and to chiropractor Sam Homola’s excellent critique of the study. My friend changed his mind and thanked me for educating him.
An example of what all too often happens
I was invited to give the “con” side of a pro/con presentation on dowsing to a local discussion group. I lent my opponent my copy of Vogt and Hyman’s classic book Water Witching USA so he would know ahead of time what I was going to say. He read it. The book explains how the ideomotor effect creates the illusion that the dowsing rod moves of its own accord and explains that dowsers have never been able to pass controlled scientific tests. I said as much in my “con” presentation. His “pro” presentation consisted of two arguments: he had personally seen dowsing work, and lots of people believed in it. He didn’t even try to rebut my facts and arguments; he simply refused to engage with them in any way. It was as if he had not read the book and had not heard anything I said. Afterwards, one of the audience was heard to say she would have liked to hear more about how dowsing worked and less about how it didn’t work!
Will Storr investigates
Sadly, some people are unpersuadable. They might as well be saying “My mind’s made up; don’t confuse me with the facts.” We have seen plenty of glaring examples in the comments section of this blog. Will Storr wrote a book The Unpersuadables: Adventures with the Enemies of Science about his struggle to understand the phenomenon. He did a great job of investigative reporting, interviewing people with strange beliefs, spending time with them and also with their critics, and reading pertinent research.
Storr found that discussing evolution with religious fundamentalists was:
…like being a tourist in another Universe… Simple facts and basic logic just don’t work the way I had assumed… facts proved entirely ineffective, and they were ineffective to a spectacular and baffling extent.
The answers they gave to his reasonable questions were often hilarious. If T. rex was a vegetarian, why did he have such big teeth? To eat watermelons.
He found that he liked the people he interviewed. They were not stupid. What made them so unpersuadable? He quickly realized that he answer was not intelligence or education or logic. Something else was going on, but what was it?
He found some clues in recent brain research.
How the brain deceives us
“I know what I saw.” No you don’t, your brain constructed an illusion. We have all seen amazing demonstrations of optical illusions, but we don’t realize we are seeing them constantly. Our brains deceive us. We don’t get the raw data from the retinal receptors; we only get the brain’s interpretation of the data. We can’t even see that we have a blind spot in each visual field corresponding to the place where the optic nerve enters the retina. The brain puts elements of vision together (horizontal lines, contrast, etc.) and constructs a best guess based on what it has learned about the world. We ignore most of our sensory input, becoming aware of things only on a need to know basis. It has been estimated that up to 90% of what you are seeing is constructed from your memories. As babies grow, they interact with the world and their brains develop internal models, ways of interpreting sensory inputs that involve shortcuts and illusions (example: if it looks smaller, it’s probably farther away). The brain builds models of what the world is like, and the senses are then used as fact-checkers, noticing anomalies that don’t match the model. If we receive information that fits well with our internal models, it is readily incorporated; if it differs, we “minimize, distort, rationalize and even hallucinate our way into disregarding this information…we lie to ourselves.”
We would like to believe that we are in conscious control of our actions and thoughts, but we are not. We have no idea why we believe what we believe. Storr illustrates this with a study where people were asked to choose the best candidate for police chief. When told the male candidate was formally educated and the woman was streetwise, they said they thought carefully and chose the man because education was more important for a police chief. When told the female candidate was formally educated and the man was streetwise, they said they thought carefully and chose the man because it was most useful for a police chief to be streetwise.
We have no idea why we do what we do. Split-brain studies and decision-timing studies show that we lack access to the unconscious processes that determine our actions and decisions, and we try to make up plausible reasons after the fact.
We can’t even trust our memories. They are reconstructed every time we access them, and they can become distorted or contaminated with other memories. Psychological studies suggest that about 30% of our memories are false, including some of the ones we are most confident about.
The brain tries to protect us by sustaining a positive self-image. When I do wrong, it was because outside circumstances conspired so I had no other option; when you do wrong it’s because of your internal character flaws. The perpetrators of evil are able to rationalize it so they can believe they are doing something good.
Errors in thinking
We are hampered by prehistoric thinking equipment. We are suckers for a well-told story; attractive plots can sucker us into false beliefs. We are tribal animals who think in binary terms. We divide people into Us vs. Them, and cast Them as villains.
Cognitive dissonance is painful; confirmation bias is comforting. Experience is re-interpreted in such a way that it doesn’t force us to rebuild our internal models of reality. We are all prejudiced, but we need prejudices to function efficiently. They serve as a practical starting point for our guesses about the world.
We are subject to confirmation bias; we get a feel-good “neurochemical kiss” as a reward for confirming a brain model. Confirmation bias serves a purpose. If we had to fairly evaluate every new argument and every bit of new evidence from scratch and constantly rebuild our models, we would become hopelessly overwhelmed and unable to function.
We are adept at ignoring information that contradicts our beliefs. Storr interviewed a woman who told him her auditory hallucinations were caused by sexual abuse she experienced at age 11-12. She then admitted she first started hearing voices at age 9. She was untroubled by the discrepancy. A Morgellon’s patient reported that his symptoms resolved when he simply stopped scratching and picking at his skin. But he still believed he had a mysterious disease that caused strange fibers to emerge from his skin.
High IQ offers no protection. A study asked people to think of as many reasons as possible for and against a belief; the higher IQ people thought of more reasons for their own beliefs, but were no better at thinking of reasons against their beliefs.
Storr visits a Nazi gas chamber with holocaust denialist David Irving. Irving notices that there are handles on the inside of the doors and says it is evidence that people were not locked in and could get out any time they wanted. He does not see that the handles are unconnected to any opening mechanism and there are bolts on the outside of the doors that would have locked the doors closed over airtight seals. What happened in his mind when he saw the bolts? Liar or deluded; evil or mistaken? Can we even hope to understand?
Probably not. Thinking is driven by emotion. Storr concludes that when Irving thinks of empire, he feels joy; when he thinks of Jews, he feels “something else.” All the rest is confabulation.
We are storytellers
Humans are storytelling animals. Our lives depend on stories. They help us make sense of the world, explaining cause and effect and helping us remember. Emotional narrative leads the news; stories that fit well with our feelings are compelling. Unfortunately, stories tend to work against truth; their purpose is not fact but propaganda. A single anecdotal experience like taking a homeopathic remedy and feeling better is interpreted by the brain and turned into an invented story about cause and effect. Science is a new kind of language that fights against the dominion of the narrative. “We can hardly be surprised if some feel an instinctive hostility towards [science], for it is fundamentally inhuman.”
Consciousness is the first storyteller. It tells us that we have free will, provides explanations for our actions, and presents self as hero. Religions and ideologies play into the hero plot since they match up well with the individual’s moral hunches and provide external justification. They validate emotional instincts, provide purpose and a common enemy. They can be useful but can also be dangerous; people have died for false beliefs.
The hero-maker is complemented by a demon-maker. Storr says that Steven Novella sees many practicing homeopaths as psychopathic con artists, while homeopathy proponent Dana Ullman sees skeptics as Big Pharma shills.
Some people accept a belief only if it can be shown to correspond to reality; others accept beliefs just because they are part of a coherent system.
If a person’s set of beliefs all cohere, it means that they are telling themselves a highly successful story. It means that their confabulation is so rich and deep and all- enveloping that almost every living particle of nuance and doubt has been suffocated. Which says to me, their brains are working brilliantly, and their confabulated tale is not to be trusted.
Storr tries to sort out the different stories told by opponents: Richard Wiseman vs. Rupert Sheldrake on his dog ESP experiments, James Randi vs. those who have accused him of lying. He interviews the individuals on both sides and does his best to find “the truth,” but all he finds is two competing stories, and he says “stories are never true.”
On skeptics
We are attracted to others whose models match our own, and in our heroic, altruistic, wonderfully human way, we go out and try to persuade others to change their models to match ours. This is certainly true of religious proselytizers, and Storr thinks it is equally true of skeptics. He makes some harsh observations about the skeptical movement, some more justified than others.
He attends skeptical conferences to observe us. At a QED conference in England, he talks to a skeptic who says homeopathy is bad but admits he hasn’t read any homeopathy studies. He thinks skeptics tend to place an unquestioning reliance on their leaders, just like the followers of any cult. At The Amazing Meeting (TAM) he overhears a non-attendee saying skeptics are like conspiracy theorists.
Storr gets some things wrong
He asks James Randi if he has ever changed his position on anything based on the evidence; Randi can’t think of anything offhand. Storr condemns Randi for not being willing to change his mind, but the fact that Randi hasn’t changed his mind doesn’t mean he is unwilling to. Storr doesn’t offer any examples of subjects about which Randi should have changed his position and didn’t. Certainly no better evidence has surfaced to make skeptics change their positions on things like homeopathy, dowsing, Nostradamus, or psychic powers. I would argue that Randi’s positions are not likely to be proven wrong by the evidence, because he based them on evidence and plausibility in the first place.
He thinks skeptics are too hard on Morgellons patients. He thinks we fall for binary thinking when we assume they all have delusions of parasitosis (DOP). He suggests that maybe some of them have an underlying undiagnosed sensory disorder that causes pruritus. I would argue that we are open to that possibility and would accept evidence that supported it, and I don’t think we are automatically labeling them all with the DOP label.
Storr thinks a lot of skeptics consider ESP so improbable that they don’t bother to read the studies and just reject them out of hand. Perhaps some do, but most don’t. Skeptic Susan Blackmore certainly didn’t; she believed in psi, studied it herself, and even got a PhD in parapsychology before she was eventually forced to give up her beliefs by the studies themselves as she watched the evidence crumble away to nothing when rigorous scientific standards and close scrutiny were applied. Skeptical psychologists like Ray Hyman and James Alcock have spent untold hours analyzing psi studies in great detail and have written about the many specific flaws they found in the studies’ methodology and data analysis. Skeptic Richard Wiseman has criticized Rupert Sheldrake’s research and even repeated one of Sheldrake’s dog ESP studies to show that his best dog subject couldn’t really sense when its owner was coming home. He clearly has an open mind; he says he is not 100% convinced that psi is wrong, only 90%.
Storr doesn’t have the same understanding of placebos as we on SBM do. He gives them more credit and says people believe bogus treatments work “because they do.” Also, he is bothered by the “hard problem” of consciousness and doesn’t seem to recognize the survival advantages of a conscious brain over an unconscious “zombie brain.”
Why do the Unpersuadables try so hard to persuade others?
Why do people have such an urge to aggressively force their views on others, to make them agree to see things the same way? Obviously, we want to confirm the models in our brain and reduce cognitive dissonance, but there is more. Storr thinks skeptics treat belief as a moral choice and fail to realize how little control we have over the ways our brains deceive us. Anyone who calls himself a freethinker betrays an ignorance of the motors of belief. The woman who believes homeopathy cured her is not free to think otherwise.
He says where there is illegality or racial hatred, call the police; where there is psychosis, call a doctor; where there is misinformation, bring learning (I see that as our role on SBM). But where there is ordinary madness and eccentricity, we should celebrate the riches of our species. “To be mistaken is not a sin. Wrongness is a human right.”
As if all this weren’t problematic enough, Storr asks a further thorny question: why do minds sometimes change? Is it because reality intrudes, or because we hear a story we like better?
Conclusion
We are creatures of illusion. To be human is to be “unpersuadable,” at least to some degree about some things. We can’t escape the limitations of our prehistoric brains, but we can learn to constantly remind ourselves that no matter how right we feel, there is a possibility that we might be wrong.

Thinking With Your Emotions About Medicine
Steven Novella on July 13, 2016
Link


The mental pathway of least resistance, what psychologists often refer to as the “default mode” of human thought, is to go with our “gut feelings.” We evolved emotions, heuristics, and cognitive biases partly so that we could make quick judgments that are good enough and err on the side of survival.
This can be adaptive – if we smell something rotten we have an emotional disgust response and avoid it. We don’t have to make a calculation about the odds of getting sick from the rotten food vs the calories and nutrients we can derive from it, we just feel disgust and avoid it.
This apparatus, however, does not deal well with a complex technological civilization that contains things like marketing and social media. The low-energy cognitive process of doing what feels right is easily manipulated and often leads us astray.
In fact I would argue that the mission of SBM can be summarized as: replacing such low-energy cognitive processes with progressively higher-energy cognitive processes, such as critical evidence-based analysis. This, however, can be a lot of work and may rub up against our feelings, and so there is often a great deal of resistance.
Marketing and health halos
Those who are trying to sell you something want you to think with your emotions, so that they can easily manipulate them. Once they successfully create an association between a feature and a positive or negative emotion, they can then market any product or service by simply saying that they have the positive features, and their competitors have the negative features.
Here are some examples:
Natural – This is the mother of all health halos, which is why we frequently discuss it. Centuries of marketing (there are examples that go back as far as there have been health products) have created a powerful cultural belief that everything natural is wholesome, healthy, safe, and effective. This is nothing more than the appeal to nature fallacy. Of course, most things in nature will kill you, or at least make you sick. The origin of something tells you nothing definitive about its features.
Organic – Some people get upset that the scientific term “organic” (referring to the chemistry of carbon) was appropriated to mean something entirely different, a system of farming that is based in the appeal to nature fallacy, fearmongering about technology, and Mother-Earth spiritualism. The misuse of a term is not the problem, however, it is the deceptive marketing of the organic industry.
So pervasive is the use of the label “organic” to confer a health halo, you can now buy organic window cleaner, tobacco, and clothes.
Juice – Fruit juice (or fruit and vegetable juice) has a definite health halo, almost as if it is a magical elixir beaming with sunshine and good health. Much of this started with a deliberate marketing campaign to sell orange juice, and save a failing orange industry in the 1920s. Orange juice was sold for its vitamin content, but also as a cure for acidosis, a rare blood disorder that was blamed for everything from cancer to fatigue.
Juice, however, is no more nutritious than food in other forms, and fruit juice is high in sugar and so is not a good choice for those trying to manage their weight.
Vitamins – Vitamins, of course, are real and a very necessary and important part of our diet. There are genuine vitamin deficiencies and conditions that need to be treated with supplementation. The marketing-based health halo, however, is the notion that all vitamins are good all the time, that more is better, and that most people should engage in routine supplementation. There is no rationale or evidence to support this position, however (and there may even be risks).
Superfood – There is no such thing as a superfood. The concept was entirely invented to market ridiculous sources of food to a public thinking with their emotions. The notion is that some foods are so tightly packed with critical nutrients that even small amounts of them will have fantastic health benefits. Algae has been sold as a superfood, but often exotic fruits are the target (like açai berry).
Often the alleged superfoods have less nutrients than more common (and cheaper) foods, like strawberries or broccoli.
Exotic – Often exotic foods, from other parts of the world, are given a health halo simply because they are exotic. This is partly the scarcity heuristic – we tend to value things based on how difficult we think they are to obtain. The farther you have to travel, or the more isolated or limited the supply, then the more valuable it must be. Common fruits and vegetables are common, so they have to be less healthy than a berry that is only found on one tropical island.
Ancient – This is the appeal to antiquity fallacy, which is the notion that if a belief or claim has been around for a long time it must be true. This is often sold with the notion that the ancients were closer to nature (the “native” heuristic), or had secrets now lost to the modern world. There is also the assumption that bad ideas cannot survive for a long time, but history shows how untrue this is. Bloodletting survived for over two thousand years, and still exists in some forms today.
Herbal – Herbs are drugs. They are dirty, poorly regulated drugs with often unknown properties. Yet, by being herbal they are supposed to be magically safe and effective. This is even taken to the extreme in some cases, with the spiritual belief that God or Gaia created herbs specifically to be Nature’s medicine for humankind.
Scientific – Yes, science still has a substantial health halo. People respect science and want it to deliver them effective health products and treatments. The problem comes with using science as a marketing health halo, rather than an actual rigorous process. This, of course, leads to pseudoscience.
The other side of the halo
The flip side of the health halo is the negative association used for fearmongering, which is useful for scaring people away from a competitor.
Chemical – The false idea is that chemicals are all artificial and therefore risky or bad for your health (chemophobia). Of course, everything is a chemical (even water), and the source of a chemical says nothing about its effects on the human body.
Toxin – Toxins are closely related to the chemical fearmongering, but a toxin is a chemical that is especially bad. What toxin fearmongering misses, however, is that the dose makes the poison. Everything is safe at a low enough dose, and everything is a toxin at a high enough dose.
Genetically modified – This is a boogeyman invented almost entirely by the organic food industry and advocates of “natural” products. The term is often vague, as there are lots of methods for altering the genes of the foods we eat. There is also no reason to think that any methods are inherently more risky than others. The end product is all that matters.
Big whatever – If a large industry makes a profit off a product, it must be bad. There is definitely some truth to the notion of being skeptical of any industry looking after its own bottom line. Where the emotional thinking comes in is making false distinctions between some industries and others, demonizing some and giving others a free pass. There is no more reason to be skeptical of Big Pharma than there is Big Supplement, Big Organic, Big Herbal, Big Homeopathy, or any other multi-billion dollar industry selling health products. Often, in fact, these industries are intertwined. The folksy mom-and-pop image of some industries is just more marketing.
Conclusion: Fear is the mind killer
There are more buzzwords and marketing ploys than those I covered here, but these are the most common ones. These are all attempts at getting you to think with your gut, to feel good about one product and bad about a competitor’s product.
Don’t let yourself be manipulated, however. Think with your brain, and with that part of your brain that requires a lot of mental effort – careful and objective critical analysis. It’s worth the effort.

   Playing by the Rules
Harriet Hall
Link
I recently read Flock of Dodos (no relation to the movie of the same name). It’s a hilarious no-holds-barred send-up of the lies and poor reasoning of the intelligent design movement. I was particularly struck by a quotation from William Dembski’s Intelligent Design.
We are dealing here with something more than a straightforward determination of scientific facts or confirmation of scientific theories. Rather we are dealing with competing world-views and incompatible metaphysical systems.
That cuts to the essence of what we are confronting on Science-Based Medicine. We are trying to evaluate the science behind claims that are often not based on science but on beliefs that are incompatible with science. The claimants are happy to use science when it supports them, but when it doesn’t they are likely to unfairly critique the science or even to dismiss the entire scientific enterprise as a “materialistic world view” or “closed-minded.” We are talking at cross purposes. How can we communicate if we say “this variety of apple is red” and they insist “it feels green to me”?
We get frustrated when we show these folks the scientific evidence and they refuse to accept it. Homeopathy is not only implausible, but it has been tested and has failed the tests. Yet proponents refuse to acknowledge those failures and still want to talk about data from the 19th century and make claims for the memory of water. We have to realize we are not even speaking the same language. We are trying to play a civilized game of gin rummy and they are dribbling a basketball all over the card table. Before getting into a debate, doesn’t it make sense to define what game you’re playing and what the rules are?
Science has been a very successful self-correcting group endeavor. It wouldn’t be successful if it didn’t follow a strict set of rules designed to avoid errors. [Note: there are no rules written in stone; I’m talking about conventions that are generally understood and accepted by scientists, conventions that grow naturally out of reason and critical thinking.] If proponents of alternative medicine want to play the science game, they ought to play by the rules. If they won’t play by the rules, they effectively take themselves out of the scientific arena and into the metaphysical arena. In that case, it is useless for us to talk to them about science.
If you want to play the science game, here’s what you do:
     1.  Submit your hypothesis to proper testing. Testimonials, intuitions, personal experience, and “other ways of knowing” don’t count.
     2.  See if you can falsify the hypothesis.
     3.  Try to rule out alternative explanations and confounding factors.
     4.  Report your findings in journal articles submitted to peer review.
     5.  Allow the scientific community to critique the published evidence and engage in dialog and debate.
     6.  Withhold judgment until your results can be replicated elsewhere.
     7.  Respect the consensus of the majority of the scientific community as to whether your hypothesis is probably true or false (always allowing for revision based on further evidence).
     8.  Be willing to follow the evidence and admit you are wrong if that’s what the evidence says.
If you want to play the science game, here are some of the things you don’t do:
     1.  Accuse the entire scientific community of being wrong (unless you have compelling evidence, in which case you should argue for it in the scientific journals and at professional meetings, not in the media).
     2.  Design poor-quality experiments that are almost guaranteed to show your hypothesis is true, whether it really is or not. Use science to show THAT your treatment works, not to ask IF it works.
     3.  Keep using arguments that have already been thoroughly discredited. (The intelligent design folks are still claiming the eye could not have evolved because it is irreducibly complex; homeopaths are still claiming homeopathy cured more patients than conventional medicine in the 19th century epidemics).
     4.  Write books for the general public to promote your thesis – as if public opinion could influence science!
     5.  Form an activist organization to promote your beliefs.
     6.  Step outside the scientific paradigm and appeal to intuition and belief.
     7.  Mention the persecution of Galileo and compare yourself to him.
     8.  Invent a conspiracy theory (Big Pharma is suppressing the truth!).
     9.  Claim to be a lone genius who knows more than all the other scientists put together.
     10.  Offer a treatment to the public after only the most preliminary studies.
     11.  Set up a website to sell products that are not backed by good evidence.
     12.  Refuse to admit it when your hypothesis is proven wrong
Changing Our Minds
Scientists will change their minds when the evidence warrants. Before we waste time arguing, one thing we can do is ask our opponents what it would take to change their minds. One woman I asked said no amount of evidence could change her mind because she knew from personal experience that her claim was true, so any evidence that said otherwise would have to be false and fabricated. End of discussion. She’s out of the game.
The rules of science are pretty clear about what it takes to change our minds. I’ll use the example of Helicobacter and ulcers. We used to think that stress and too much stomach acid caused ulcers; now we think a bacterium causes ulcers. Here’s a summary of why we changed our mind:
     1.  Scientists noticed bacteria in biopsy samples from ulcers.
     2.  They identified the bacteria as Helicobacter pylori
     3.  They found a strong correlation between ulcers and the presence of the bacteria
     4.  One of the researchers who was healthy and not a Helicobacter carrier was able to induce an ulcer in himself by ingesting the bacteria.
     5.  They found that treating patients with antibiotics cured ulcers.
     6.  They found that antibiotics were superior to previous ulcer treatments.
     7.  The studies were replicated and studies done in different ways corroborated each other.
     8.  The bacterial hypothesis was not inconsistent with the rest of scientific knowledge.
If we had that quantity and quality of evidence for homeopathy, we’d gladly accept it. In fact, if we had 1 through 7 we’d provisionally accept it while we kept checking the data and tried like crazy to figure out the mechanism behind it.
There are two issues that are often misunderstood: scientific consensus and prior plausibility.
Prior Plausibility
Homeopathy is completely implausible. We would have to accept robust evidence that it worked, but we would require much stronger evidence than we would for, say, a new antibiotic. Because if the claims for homeopathy were true, we would have to revise much of what we know about physics, chemistry, and physiology.
The crossword analogy is helpful. If you think the answer to 1-across should be “library” but the clue to 1-down is a five-letter word for the author of Tom Sawyer and the clue to 2-down is a four-letter word for the name of Eve’s husband in Genesis, you have to reject “library” and keep looking for a word that starts with T-A. You have to recognize that no matter how strong your conviction that 1-across must be “library” you must be wrong and that there must be another answer that you just haven’t thought of.
Consensus
It’s easy to dismiss the scientific consensus as a popularity contest, a vote on opinions. But it’s far more than that. The body of evidence stands or falls on its own merits, and when the weight clearly tips the balance to one side everybody can see it. The scientific community is made up of experts who know how to evaluate the evidence and who thrash out disagreements in medical journals and scientific conferences. It is easy for the scientific community to reach an agreement based on clear evidence. There are times when the outcome is uncertain and controversy among scientists is appropriate, but there comes a time when it would be perverse not to accept the evidence, just as it is perverse to deny evolution or germ theory. The scientific consensus on evolution and the germ theory is a recognition of reality, not a matter of opinion.
A reasonable default assumption is that the scientific consensus is usually right, and if it isn’t, it will change as the evidence becomes clearer. Truth will prevail. It does no good to attack the scientific consensus as prejudiced or closed-minded. The consensus will only change when it incorporates new and better evidence. One of the irrational tactics we’ve seen over and over on this blog is for opponents to cite one or a handful of studies to support their belief, in the ridiculous assumption that it was new information that the people who reached the scientific consensus had failed to consider, or that it somehow outweighs all the other studies that found the opposite.
Play by the Rules or Go Play Your Own Game
There’s no point in arguing scientific facts with someone whose world-view is metaphysical and non-scientific. There’s no point in presenting geological age data to someone who “knows” the age of the Earth from the Bible. Before we get into a useless debate, maybe we should find out what game our opponents are really playing. If they are playing pingpong, it’s silly for us to bring a football to the table. It would be handy if we could get them to say up front what game they are really playing, but all too often they have deluded themselves into truly believing they are following the rules of science.
If they won’t play the science game by the rules, we are justified in crying “foul” and disqualifying them. Then they can go away somewhere else and play their own game by whatever rules they want and we won’t be able to refute them. If they are relying on beliefs unsupported by evidence, let them say so. Wouldn’t it be refreshing to hear a homeopath say “I believe homeopathy works based on my personal experience and on non-scientific evidence like testimonials, and I categorically reject the results of any scientific trial that fails to support my beliefs. Homeopathy cured my neighbor’s uncle’s cousin of cancer. Trust me. I’m a nice guy so you should believe whatever I tell you.”
If they’d say that up front, we wouldn’t waste any of our valuable time re-hashing scientific evidence that they would just ignore. They would be out of the game, permanently. And patients would have a better basis for giving truly informed consent

   On Being Certain
Harriet Hall
Link
Neurologist Robert A. Burton, MD has written a gem of a book: On Being Certain: Believing You Are Right Even When You’re Not. His thesis is that “Certainty and similar states of ‘knowing what we know’ arise out of involuntary brain mechanisms that, like love or anger, function independently of reason.” Your certainty that you are right has nothing to do with how right you are.
Within 24 hours of the Challenger explosion, psychologist Ulric Neisser had 106 students write down how they’d heard about the disaster, where they were, what they were doing at the time, etc. Two and a half years later he asked them the same questions. 25% gave strikingly different accounts, more than half were significantly different, and only 10% had all the details correct. Even after re-reading their original accounts, most of them were confident that their false memories were true. One student commented, “That’s my handwriting, but that’s not what happened.”
Just as we may “know” things that clearly aren’t true, we may think we don’t know when we really do. In the phenomenon of blindsight, patients with a damaged visual cortex have no awareness of vision, but can reliably point to where a light flashes when they think they are just guessing. And there are states of “knowing” that don’t correspond to any specific knowledge: mystical or religious experiences.
A “feeling of knowing” probably had an evolutionary advantage. If we are certain, we can act on that certainty rather than hesitating like Hamlet. Certainty makes us feel good: it rewards learning, and it keeps us from wasting time thinking too much; but it impairs flexibility.
Richard Feynman said,
“I can live with doubt and uncertainty and not knowing. I have approximate answers and possible beliefs and different degrees of certainty about different things… It doesn’t frighten me.”
On the other hand, many people, especially religious fundamentalists, can’t deal with uncertainty. They demand absolute answers and cling to their certainties even in the face of contrary evidence. Why are people so different in their need for certainty? We know there is a gene associated with risk-taking and novelty-seeking. Burton makes an intriguing suggestion: could genetic differences make individuals get different degrees of pleasure out of the feeling of knowing?
There is a “hidden layer” in our brain whose neurons are influenced by genetics, personal experience, hormones, and chemistry. These factors influence all our thought processes without our conscious knowledge. We would like to think that if everyone had the same information they would necessarily reach the same conclusion, but that just isn’t so. There is no such thing as pure reason. “Reason is not disembodied, as the tradition has largely held, but arises from the nature of our brains, bodies, and bodily experiences.”
The autonomous rational mind is a myth. The concepts of the self and free will are innate useful fictions that allow us to function. As Samuel Johnson said, “All theory is against the freedom of the will; all experience is for it.” Modern neurophysiology tells us our decisions are made subconsciously before we are aware of deciding.
Burton discusses how certainty interferes with science. “Integrative medicine” guru Andrew Weil set up tests of osteopathic manipulation for ear infections, and when the experiments showed no effect, he said, “I’m sure there’s an effect there. We couldn’t capture it in the way we set up the experiment.” This kind of thinking is rampant in alternative medicine. Burton thinks that if Dr. Weil recommends osteopathy for an ear infection, he should inform the patient that the recommendation is based on an unconfirmed belief.
Richard Dawkins rejects religion but finds purpose and meaning in science. Burton suggests that purpose and meaning are powerful innate feelings. We feel that our life has purpose and meaning, and we look to science or religion to try to explain that feeling. No amount of rational argument is likely to change us. “Whether an idea originates in a feeling of faith or appears to be the result of pure reason, it arises out of a personal hidden layer that we can neither see nor control.”
Burton thinks irrational beliefs can have adaptive benefits (for instance, the placebo effect) and thinks objectivity and reason should be seen in the larger context of our biological needs and constraints. If science and religion could both accept that all our facts are really provisional, absolutism could be dethroned and a dialog might become possible. What if religious fundamentalists acknowledged even a 0.0000000001% possibility that their beliefs were false? Biology teaches us that absolutism is an untenable stance of ignorance.
I have long thought that absolutism was one of humanity’s greatest problems. There are implications for politics, religion, and every sphere of human activity. The insights from this book can be applied to every human interaction from marital squabbles to terrorism. It may be frightening to recognize the limits of our knowledge. It will be hard for some to give up their cherished certainties, but Burton says he has gained an extraordinary sense of an inner quiet born of acknowledging his limitations.
As a reminder that there is never a 100% guarantee that we are right, Burton suggests we use the words “I believe” instead of “I know.” This is the one place where I disagree with him: I don’t like either word. Belief sounds too much like faith. I don’t like the idea of saying I believe evolution is true. Truth in science, at best, can only mean that the evidence is overwhelming. We can’t “know” absolutely in a metaphysical sense. We provisionally accept evolution because the evidence is so overwhelming that it would be perverse to reject it. We remain open to new evidence.
The author is a neurologist who is also a novelist and a columnist for Salon.com. This well-written book is the result of many years of cogitation by a wise clinician. He supports his arguments with tales of neurology patients, recent research into brain function, and examples of how our senses constantly fool us.
Burton says, “In medicine, we are increasingly developing ethical standards for complex medical decisions that both allow for hope and placebo effect, yet don’t fly in the face of evidence-based medical knowledge.” This subject has come up on this blog before, and it is one we will continue to grapple with.
If there’s anything you think you’re certain of, read this book and you may change your mind.

The Brown M&M’s of Science-Based Medicine
Scott Gavura on April 21, 2016
Link


Medicine is constantly changing, and like most health professionals, I am required to maintain my competency to practice. I doubt pharmacists are unique in being inundated with offers of continuing medical/pharmacy education. Some courses are free, some cost hundreds of dollars, and it can be difficult to distinguish the high-quality programs from the biased or low-quality education that furthers a agenda, rather than seeking to truly educate. You can consider the reputation of the provider, or the author, and sometimes the sponsorship gives a clue. When it comes to determining if a program’s content is science-based or not, I find the learning objectives may be all I need to read. One program I saw recently referred to “integrative” approaches to the treatment of an illness. Another claimed it would teach you a “holistic” approach to managing complex medical condition. Both programs set off skeptical alarm bells. I realized then I’d found the science-based medicine equivalent of a brown M&M. And I have the band Van Halen to thank for that association.
No brown M&Ms
I’m a child of the 1980s. Neon fashions. Big hair. It was an era when rock music, preferably played on cassette, on a boom box, was king. And for much of the 1980s, there probably no bigger rock bank than Van Halen. With the amazing guitar work of Eddie Van Halen and the showmanship of lead singer David Lee Roth, Van Halen made heavy metal party music. The album that catapulted Van Halen to 80’s rock gods was their album, 1984. It was a smash hit. The videos were outrageous. Van Halen was pretty much every 1980s rock caricature you could think of. They rocked hard, they partied harder.
Van Halen concerts filled stadiums and were over-the-top, onstage and backstage. As one of the world’s biggest groups at the time, a Van Halen concert was a massively complex sound and lighting show, requiring a fleet of transport trucks to carry everything from city to city. True to their reputation, their list of demands from concert promoters grew and grew. Contracts had dozens of riders, outlining not only highly technical requirements for the shows but a long list of perks for the performers. Van Halen apparently had a reputation for making absurd demands – because they could. Buried deep in the concert rider was one demand that seems too ridiculous to be true.
There were stories of what happened when promoters didn’t follow this request. David Lee Roth, if he found a bowl of M&M that didn’t meet their specifications, had a reputation for trashing dressing rooms. And while this might sound like the height of rock star ego, there was a very specific reason for burying a demand for M&M’s deep in the back of a concert contract.
It was a quality control measure.
The M&M demand had been placed there deliberately by the band to ensure that the promoter had read through the entire contract and fulfilled it in its entirety. If the request had been ignored, the band apparently insisted on a complete check on all other aspects of the production. From David Lee Roth’s biography, via Snopes:
So, when I would walk backstage, if I saw a brown M&M in that bowl … well, line-check the entire production. Guaranteed you’re going to arrive at a technical error. They didn’t read the contract. Guaranteed you’d run into a problem. Sometimes it would threaten to just destroy the whole show. Something like, literally, life-threatening.
Rather than representing enormous rock star ego, the M&M clause was a quick and visible test that verified the promoter was taking the technical requirements seriously.
The brown M&M’s of Science-Based Medicine
Back to science and medicine. The SBM equivalent of a brown M&M is anything that is a warning sign for pseudoscience, or simply a lack of critical thinking. It’s a sign you need to dig deeper. If I was considering registering for a continuing education program in HIV or transplant (two areas that I lack any current experience in), I lack the expertise to determine if the content is of high quality. But if that program makes a positive reference to “complementary and alternative” approaches, then that’s a signal I need to look more closely and critically at all of the content that’s presented.
In hindsight, I realize I’ve done a form of the M&M test with health professionals, too. Years ago I was looking for a family doctor and met with a physician who was known to the local pharmacy I worked at. After some general discussion of health issues, I asked him for his opinion of homeopathy, since I’d noticed many of his patients tended to use it. His response was optimistic – he’d “seen results.” I realized on the spot that I didn’t want him as a family doctor, because if he was fooling himself with homeopathy, I didn’t think I could trust him with guiding my health care decision-making.
You can try a similar test with your pharmacist. Most retail pharmacists have no input into what’s sold on a pharmacy’s shelves (a topic for another day). But seeing something like homeopathy on the shelf is an excellent opportunity for dialogue. Ask them if homeopathy works. If the answer is anything but “placebo effects”, then that’s your brown M&M. If a pharmacist isn’t frank with you about the ineffectiveness of homeopathy, or hasn’t taken the time to understand the science, then you might want to rethink the pharmacy you support.
Here’s an incomplete list of words that, in the right context, are the SBM equivalent of an brown M&M. The mere appearance of any of these words doesn’t necessarily mean there’s a problem. But it’s a signal that you need to proceed carefully:
    • biomedical
    • holistic/wholistic
    • integrative/integrated (though the latter is preferred over the former)
    • complementary and alternative
    • “functional” medicine
    • natural
    • quantum
There are probably dozens more – list any of your triggers in the comments, and I’ll add them to the list.
Conclusion: Quality health care is in the details
Details matter. Science matters. All of the contributors to the blog donate their time for the same reason: We want to promote good medicine. Pseudoscience compromises and pollutes good medical care. Rock band Van Halen inserted an absurd food demand deep in a concert rider. It was easy to ignore. Brown M&M’s were their “red flag” for potentially more serious problems that might otherwise go unnoticed or ignored – until they caused harm. For that same reason we regularly document the problems of pseudoscience that masquerades as medicine. Homeopathy may seem harmless until someone dies from substituting it for real medicine. “Complementary” and “alternative” approaches to health may sound low-risk until it leads to ignoring real medicine. The best medicine means a relentless focus on the quality of care. If health professionals don’t take the science and evidence standards of that care seriously, who will?

   Ambiguity
Mark Crislip
Link
Some people have made the mistake of seeing Shunt’s work as a load of rubbish about railway timetables, but clever people like me, who talk loudly in restaurants, see this as a deliberate ambiguity, a plea for understanding in a mechanized world. The points are frozen, the beast is dead. What is the difference? What indeed is the point? The point is frozen, the beast is late out of Paddington. The point is taken. If La Fontaine’s elk would spurn Tom Jones the engine must be our head, the dining car our esophagus, the guard’s van our left lung, the cattle truck our shins, the first-class compartment the piece of skin at the nape of the neck and the level crossing an electric elk called Simon. The clarity is devastating. But where is the ambiguity? It’s over there in a box. Shunt is saying the 8:15 from Gillingham when in reality he means the 8:13 from Gillingham. The train is the same only the time is altered. Ecce homo, ergo elk. La Fontaine knew his sister and knew her bloody well. The point is taken, the beast is moulting, the fluff gets up your nose. The illusion is complete; it is reality, the reality is illusion and the ambiguity is the only truth. But is the truth, as Hitchcock observes, in the box? No there isn’t room, the ambiguity has put on weight. The point is taken, the elk is dead, the beast stops at Swindon, Chabrol stops at nothing, I’m having treatment and La Fontaine can get knotted.
— Art Critic
Ambiguity. Medicine, like art, is filled with ambiguity, at least the way I practice it. Most of my practice is in the hospital. I am sometimes called to see patients that other physicians cannot figure out. And that puts me at a disadvantage, because the doctors who were referring patients to me are all bright, excellent doctors. Often the question is ‘Why does the patient have a fever?’ or ‘Why is the patient ill?’ Sometimes I have an answer. Most of the time I do not.
I am happy, however, to be able to tell the patient what they don’t have. I can often inform the patient and their family that whatever they have is probably not life-threatening or life-damaging, just life-inconveniencing, and most acute illnesses go away with no diagnosis. I always put the ‘just’ in air quotes, because illnesses that require hospitalization are rarely ‘just.’ Just without quotes is reserved for the antivaccine crowd and applied to the small number of deaths from vaccine preventable diseases in unvaccinated children. John Donne they ain’t.
We are excellent, I tell them, at diagnosing life-threatening problems that we can treat, and terrible at diagnosing processes that are self-limited. Of course diagnostic testing is always variable. No test is 100% in making a diagnosis, and often with infections I cannot grow the organism that I suspect is causing the patient’s disease. So for hospitalized patients, ambiguity and uncertainty are the rule of the day.
However, the situation is much better than they used to be. I am now one of the oldest physicians practicing in my hospital. After 21 years most of the prior old guard has retired or died, leaving me. I have gone from being the young whippersnapper to the old geezer in what to me seems to be a blink of an eye.
However, the advantage to being old is you get to bore people with the stories of your gloried past. I remember a time, I tell the residents, before CAT scans, before third-generation cephalosporins, before PCR diagnostics. I remember the beginning of the AIDS epidemic, when we saw young men dying of an unknown illness. I still vividly remember my first AIDS patient, dying from disseminated MAI, who offered me a chocolate from his box of candy. I declined. I told him I wasn’t hungry. He told me “I would have to spit in your mouth to give you AIDS.” I did not know that at the time. No one did. Today I would eat the candy.
Times have changed, mostly for the better. AIDS has gone from an unknown disease with a short life expectancy to a mostly chronic manageable illness whose pathophysiology is understood in remarkable detail. Medicine advances. It is often an uncomfortably slow and aggravating process, because diagnostics and therapeutics that look promising at the beginning often turn out to not live up to their promise. Kind of like many people I have known.
Some therapeutic interventions have remained in limbo my entire practice. Steroids, as an example, have been tried for every illness except for Cushing’s disease. In almost every instance they have been found wanting. When I was an intern, every patient with a neurologic event was put on aspirin and Persantine. I don’t think Persantine is used much anymore. Common admission diagnoses were an aminophylline toxicity and digoxin toxicity; both drugs are rarely used today since we have less toxic and superior alternatives.
Certainly my practices changed dramatically over the last 21 years. I used to make a living from diseases that are rapidly becoming of historical interest. Ventilator-associated pneumonia, line-related sepsis, AIDS opportunistic infections, neutropenic fever’s, diabetic foot infections in smokers, all used to be common admitting diagnoses that resulted in infectious disease consultation. No longer.
Despite the wackaloon opinion that doctors are in it for the money, combined with big pharma greed, the last 21 years has seen a concerted effort on the part of the medical industrial complex to decrease the diseases we treat for living. This is not only true in infectious diseases, but cardiologists have been at the forefront of stop smoking and lipid control. The same is true of pulmonary doctors. Every physician fights the battles of obesity in the outpatient clinic. Much of the time physicians try to put themselves out of work. And in infectious diseases it seems to be successful. That is why at TAM 9 this year I plan on letting everyone buy me a beer. I’m sure the rest of the SBM crew would feel the same way. But no lite beers, puh-leaze.
Medicine does advance.
There is an infectious disease therapy that superficially resembles acupuncture and homeopathy: ribavirin. A drug with few proven benefits. Like most SCAMs, case reports, uncontrolled series and wishful thinking has kept ribavirin alive and around for my entire practice. I say superficially as most SCAMs now have a proven lack of benefit. Ribavirin is an antiviral medication that has probably been tried on virtually every virus, but has never been shown to have efficacy by itself in almost any infection. It is of benefit in RSV, and combined with interferon for the treatment of hepatitis C.
Ribavirin is a broadly active antiviral has rarely been tested in randomized controlled trials. Many of the infections that are allegedly treated with ribavirin are not common in the United States. So when a question of West Nile virus, dengue virus, Tick-borne Encephalitis Virus, Yellow Fever Virus, Lassa fever, Crimean-Congo hemorrhagic fever, or Hantavirus appears, the answer is ribavirin. But is the answer the correct?
This leads to an interesting editorial from several years ago in the journal Clinical Infectious Diseases entitled How Medicine Advances. How?
The editorial concerned an article on a study that looked at the efficacy of ribavirin in the treatment of Japanese encephalitis virus. Significant time, money and effort has been expended using ribavirin for diseases like Japanese encephalitis. But there have never been randomized clinical trials to demonstrate or deny the efficacy of ribavirin in the treatment of Japanese encephalitis. Until 2009 that is.
In CID that year they published a randomized placebo-controlled double-blind study that evaluated the effectiveness of ribavirin in the treatment of children who had Japanese encephalitis. And ribavirin was found to do nothing.
What was striking about this trial, as pointed out in the editorial, was that the study was done in the poorest part of India, it was done in children, and it was done with a definitive rigor that allowed the issue of ribavirin (always with the caveats of orally and at the dose given) to be put to rest for the treatment of this one infectious disease. A little more ambiguity in medicine has been removed.
I think the final paragraphs of the editorial sum up nicely why we do science-based medicine and the importance of doing clinical trials to determine what does and does not wor:
Kumar et al., whose study is published in this issue of Clinical Infectious Diseases, are to be commended for refusing to bow to any of the complexities reputed to make clinical trials impossible. In Uttar Pradesh, India’s most populous and poorest state, Kumar and colleagues sustained over 3 years the first randomized, placebo-controlled, doubled-blind trial of ribavirin for the treatment of the most vulnerable patients—children (age, 6 months to 15 years)—to be hospitalized with acute febrile encephalopathy, and they per- formed seroreactive testing for IgM anti-bodies to Japanese encephalitis virus. By so doing, they established that oral ribavirin, at the dosage used in their study, did not improve either early or late outcomes. By demanding scientific justification for investment in this mode of therapy, they have both encouraged searches for more-effective interventions and prevented the expenditure of scarce resources ineffectively.
Both faith and science are important components of the art of medicine. We ought not to mistake one for the other.
I wish, besides sarcasm punctuation marks, we had whiny little baby tags punctuation marks, since the lament of many a SCAM proponent is that their particular intervention can’t be tested because of “complexities reputed to make clinical trials impossible.” Riiiiggghhhhttttt, Mr. Powers. It is easier to curse the darkness than to light a candle.
There is then the more difficult application of applying the data. If someone has a long history of being committed to a treatment, it is surprisingly difficult to get individuals and groups to alter their behavior. I expect the urge to give ribavirin for Japanese encephalitis will rapidly fade. Not so the urge to balance qi, fix subluxations, or realign the energy flux. Wait. The last is either reiki or Galaxy Quest. The latter at least is recognized as fiction. Unfortunately, there are many other infections for which people will try ribavirin and for which there are no randomized placebo-controlled clinical trials. Ribavirin will continue to be a drug mostly searching for a disease.
But still medicine progresses. Studies get done, there is an incremental improvement in our understanding of the diagnosis or treatment of the disease. And slowly and painfully medicine changes. Emphasis on slowly. Change has to be balanced with the knowledge that much of the information we have in medicine is not final. When I talk about studies with residents, I try and be careful to mention the often endless caveats about the applicability of the results beyond the study population. Back in the day it seemed that all coronary artery disease studies were done in old, white male veterans. Probably widely applicable, but maybe not. But at least as an old white man, I can be taken care of.
There’s an old saying that goes something like ‘be neither the first to abandon the old nor be the first to use the new.’ I certainly feel that way about antibiotics. Over the years new drugs have been approved, released into widespread use, and then found to have serious side effects that resulted in their being withdrawn from the market. So I’m always a little leery about new medications and new treatments unless I do not have other options.
So I look back on 21 years of infectious diseases 25 years of being a physician and note the incredible changes that have occurred. Diagnostics that have improved, therapies that have improved, and more importantly diagnostics and therapies that have been abandoned. Abandoned because they were shown not to work. Medicine advances.
Contrast that with the bête noire of this blog: supplements, complementary and alternative medicine. Anybody who subscribes to reality-based medicine would say at this point that the preponderance of data strongly points to the conclusion that most SCAMs do not work. Acupuncture, homeopathy, energy medicines, etc. do not materially alter disease. Yet has any of these ever been abandoned? Nope.
It would seem that they are being embraced, at least in academic institutions. SCAMs are an archetype example of failing up. It has been noted that with SCAMs that better and better studies show less and less effect until well-designed studies show no effect. For the last decade it would seem the greater the failure, the greater the spread into academia and the more popular the SCAM. By the same standards we should be using internal mammary artery ligation for coronary artery disease, high dose chemotherapy with bone marrow transplant for breast cancer, and continue to suppress all abnormal cardiac rhythms in heart attack patients. All the interventions failed spectacularly, and so should be embraced, each with their endowed Chairs.
SCAMs probably are growing for the financial benefit. Since standard medicine has declining reimbursement and most alternative therapies are out-of-pocket, it’s a good cash cow for institutions that want a flow of money and are not picky about their intellectual standards. Not only are the standard SCAMs proliferating, they often combine in most peculiar ways to come up with new variations. Doctor Moreau would be impressed with the slow mutant reassortments: acupuncture (at least 6 kinds) morphing into acupressure, laser acupuncture, acupuncture with tuning forks and color, and soon there will be dark energy acupuncture. You heard it here first. Don’t get me wrong, I am jealous. I would love to combine ID with cardiac bypass surgery and make some real cash, but they just don’t mix. Sigh.
It is said that the majority of medical practice has no basis in science-based medicine. Certainly in the practice of infectious diseases in the hospital, that is often the case. I will see an organism in a odd place, for example a Gemella endocarditis, and there are no long-term randomized placebo-controlled clinical trials to determine what the best therapy is for the treatment of a Gemella endocarditis. There probably never will be. It is so rare that it is probably impossible to generate enough cases to do a clinical trial. I am stuck with the basic principles of biologic plausibility and in vitro antibiotic susceptibilities and that is often enough. I know that if I can kill the bug in the test tube, I can often kill it in the patient as well. In the absence of clinical trials, reality can reliably determine effective therapy.
It is quite a stark contrast between SCAMs and of medicine and how they are practiced. Medicine changes. Or perhaps it would be better to say medicine evolves. The old is be shown to be worthless. It is abandoned, and patient care overall improves. Even when there are no good clinical trials to guide therapies, often we have prior plausibility and biologic plausibility to help guide our therapies. Not always, as ribavirin demonstrates, but we have to fight the war with the armies we have. Advances have not been without their side effects and bad consequences, as no good deed ever goes unpunished, but medicine still adheres to the Victorian principle that human societies are perfectible. And while we always fall short of our goal, what has accomplished has been admirable.
SCAMs persist with no improvement, no evolution, and are increasingly discredited by reality. Nothing is ever abandoned, instead persisting, mutating and growing. With alternative memes, what determines replicative fitness is not, apparently, the real world. Oh well, it gives me something to write about.

What Should We Do in the Absence of Evidence?
What to do in the absence of a clear diagnosis and randomized, controlled trials? Often nothing, sometimes something. It’s complicated.
Mark Crislip on May 1, 2015
Link

For every complex problem, there is an answer that is clear, simple—and wrong.
– H.L. Mencken
Despite my multiple personalities, it seems that only the OCD doctor gets anything done. The Goth cowgirl persona? Lazy. And the NBA playoffs are sucking up an inordinate amount of time. Go Blazers. Just not very far. Sigh. But what are you going to do. Work needs doing and someone has to do it.
This week was one of deadlines. In June I am giving a series of talks at the SMACC conference in Chicago and I have to have all my talks ready to go today. So sometimes to meet all my deadlines I need to re-purpose other material.
Spoiler alert: if you are going to be at SMACC and hear my lectures, stop reading here. Everything I am going say in 6 weeks will follow. And really even if you are going to SMACC, it is a content-free post. You might be better off spending your time elsewhere.
My topic is “What Should We Do in the Absence of Evidence?” It is often said that only 15% of medical practice is evidenced based. It is a myth, and an outdated myth at that, based on a survey from 1961. Like all myths it has staying power and people like to repeat the factoid without bothering to see if it is actually true. It isn’t.
The real data suggests we do a pretty good job at following evidence as long as we are not TV doctors:
Thus, published results show an average of 37.02% of interventions are supported by RCT (median = 38%). They show an average of 76% of interventions are supported by some form of compelling evidence (median = 78%).
There is variability depending on the specialty and how you judge the evidence. Anesthesiologists do the best, but then they have the least to do.
There is always evidence. There are more than 23,000,000 citations on Pubmed and 799,000,000 hits searching ‘medicine’ on Google. It is not a lack of evidence that is the problem. It is more likely that the practitioner doesn’t ww the evidence or the evidence may suck.
I tend to get consults for odd or unusual infections for which there is little data beyond case reports and the antibiotic susceptibility. So I am used to dealing with supporting data that is sparse at best. I often say that part of being a specialist is being ignorant with style.
So what to do.
First. Don’t panic. And carry a towel. Most of the time there is no hurry to intervene in whatever process is making the patient ill. You usually have time to sort things out. As the saying goes,
AT A CARDIAC ARREST, THE FIRST PROCEDURE IS TO TAKE YOUR OWN PULSE.
Most of the pressure comes not from the patient and her disease but the systems that do not allow the slow and methodical evaluation of disease. The old definition of a Fever of Unknown Origin, two weeks in the hospital with a fever, is a fantasy in modern medicine. Madeline would have her admission denied.
So take your time. Watch and wait. Much of the time when you do nothing, eventually the answer will become obvious.
THE DELIVERY OF GOOD MEDICAL CARE IS TO DO AS MUCH NOTHING AS POSSIBLE.
Yet another proverb:
Muddy water. Let stand. Will clear.
The “don’t just stand there, do something” that drives an intern becomes “don’t just do something, stand there” that I specialize in. And my wife can vouch for me: I am most excellent at doing nothing.
Besides, I learned as a resident that if you have no data to support an intervention, you are more likely to cause harm than to do good. I have no data to support this assertion and have never found a reference, so it is one of innumerable bits of unvalidated ‘wisdom’ I have picked up over the years. Medicine should be neither Nike (just do it) nor a Chimp (throwing crap at the wall and seeing what sticks).
Often the intervention is uncertain because you do not have a diagnosis. The best thing to be done is to retake the history, looking for hints in the pattern of disease or other risks for odd diseases. Then Google and Pubmed everything, especially the odder aspects of the case. It is amazing how often Google will come up with the right diagnosis, or at least point you in the right direction. And once you have the diagnosis, then you can proceed with the right therapy.
But be skeptical. It is easy to become enamored of a diagnosis and treatment. And do not let advertising terms guide therapy. There are only four absolutes in medicine with no exceptions. Ever. One absolute is anyone who routinely wears their scrubs outside the hospital is a dick. Another is anyone who uses “strong”, “big gun”, or “powerful” as an adjective about antibiotics is an idiot who knows nothing about the treatment of infections. They are advertising adjectives, providing false comfort but no benefit. Don’t do an intervention to make yourself feel good. It really is OK to say “I don’t know what to do” as long as you have a plan with how to proceed.
But let’s say you have a problem for which there is uncertainty as to the proper intervention. There are no guidelines or your particular patient doesn’t match the patient population in the studies. When I was a resident, all the data for cardiovascular treatments appeared to come from VA studies of old, white smoking males. Do the results apply to other populations? Maybe. Maybe not.
Now what?
You can reason from basic principles of anatomy, physiology etc. In ID it is kinda sorta easy: if the drug kills the organism in the test tube and then gets into the infected space, it should work. Other diseases are not so easy. When in doubt, should the disease get steroids? There is probably no disease, including Cushing’s Syndrome, for which steroids are considered by someone despite a lack of supporting data.
But at the end of the day you need to fire up the Chrome and go looking for the evidence, such as it may be.
So then you have to consider the published literature. What are the levels of evidence?
Eminence based medicine—The more senior the colleague, the less importance he or she placed on the need for anything as mundane as evidence. Experience, it seems, is worth any amount of evidence. These colleagues have a touching faith in clinical experience, which has been defined as “making the same mistakes with increasing confidence over an impressive number of years.” The eminent physician’s white hair and balding pate are called the “halo” effect.
Vehemence based medicine—The substitution of volume for evidence is an effective technique for brow beating your more timorous colleagues and for convincing relatives of your ability.
Eloquence based medicine—The year round suntan, carnation in the button hole, silk tie, Armani suit, and tongue should all be equally smooth Sartorial elegance and verbal eloquence are powerful substitutes for evidence.
Providence based medicine—If the caring practitioner has no idea of what to do next, the decision may be best left in the hands of the Almighty. Too many clinicians, unfortunately, are unable to resist giving God a hand with the decision making.
Diffidence based medicine—Some doctors see a problem and look for an answer. Others merely see a problem. The diffident doctor may do nothing from a sense of despair. This, of course, may be better than doing something merely because it hurts the doctor’s pride to do nothing
Nervousness based medicine—Fear of litigation is a powerful stimulus to over investigation and over treatment. In an atmosphere of litigation phobia, the only bad test is the test you didn’t think of ordering.
Confidence based medicine—This is restricted to surgeons.
There many conceptual frameworks that ranks levels of evidence in medicine:
Level Type of evidence
1A Systematic review (with homogeneity) of RCTs
1B Individual RCT (with narrow confidence intervals)
1C All or none study
2A Systematic review (with homogeneity) of cohort studies
2B Individual Cohort study (including low quality RCT, e.g. <80% follow-up)
2C “Outcomes” research; Ecological studies
3A Systematic review (with homogeneity) of case-control studies
3B Individual Case-control study
4 Case series (and poor quality cohort and case-control study
5 Expert opinion without explicit critical appraisal or based on physiology bench research or “first principles”
Which works well when dealing with reality-based processes, but as this blog has noted from the beginning, it does not work for SCAMs when there is no prior plausibility for the intervention.
Of course the volume of information makes it virtually impossible for an individual clinician to master the literature on the odd disease that may appear in the ER or on the ward. It is difficult to read and appreciate a paper above your pay grade. I can read an ID paper with reasonable comfort, but although I am board-certified in Internal Medicine, a paper on cardiology or nephrology can be difficult to place in context. I have to use rules to help me understand it:
    • Research Phrase: It has long been knownTranslation/Meaning: I didn’t look up the original reference.
    • Research Phrase: A definite trend is evidentTranslation/Meaning: These data are practically meaningless.
    • Research Phrase: Of great theoretical and practical importanceTranslation/Meaning: Interesting to me.
    • Research Phrase: While it has not been possible to provide definite answers to these questionsTranslation/Meaning: An unsuccessful experiment, but I still hope to get it published.
    • Research Phrase: Three of the samples were chosen for detailed studyTranslation/Meaning: The results of the others didn’t make any sense.
    • Research Phrase: Typical results are shownTranslation/Meaning: This is the prettiest graph.
    • Research Phrase: These results will be shown in a subsequent reportTranslation/Meaning: I might get around to this sometime, if I’m pushed / funded.
    • Research Phrase: The most reliable results are those obtained by JonesTranslation/Meaning: He was my graduate assistant.
    • Research Phrase: It is believed thatTranslation/Meaning: I think.
    • Research Phrase: It is generally believed thatTranslation/Meaning: A couple of other people think so, too.
    • Research Phrase: It is clear that much additional work will be required before a complete understanding of the phenomenon occursTranslation/Meaning: I don’t understand it.
    • Research Phrase: Correct within an order of magnitudeTranslation/Meaning: Wrong.
    • Research Phrase: In my experienceTranslation/Meaning: Once.
    • Research Phrase: In case after caseTranslation/Meaning: Twice.
    • Research Phrase: In a series of casesTranslation/Meaning: Thrice.
    • Research Phrase: According to statistical analysisTranslation/Meaning: Rumor has it.
    • Research Phrase: A statistically oriented projection of the significance of these findingsTranslation/Meaning: A wild guess.
    • Research Phrase: Thanks are due to Joe Blotz for assistance with the experiment and to George Frink for valuable discussionsTranslation/Meaning: Blotz did the work and Frink explained to me what it meant.
    • Research Phrase: A careful analysis of obtainable dataTranslation/Meaning: Three pages of notes were obliterated when I knocked over a glass of wine.
    • Research Phrase: It is hoped that this study will stimulate further investigation in this fieldTranslation/Meaning: I quit.
In the end your best bet is to call someone like me. Someone who, within the limitations of life, has a (relative) mastery of the topic. But more importantly, it then becomes someone else’s problem to figure out what to do. Good luck. You will need it.

Horses, Zebras, and the Availability Heuristic
Clay Jones on July 1, 2016
Link


During a particularly difficult shift early in my career, I spent the better part of two hours at the bedside with a patient’s family discussing the unexpected discovery of a large tumor in their child’s brain. The implications of the finding were grave, and the family was understandably devastated. I was just a few years out of residency and this was the first time I had made such a life altering diagnosis by myself, and it was by far the hardest news I had ever had to break to a family. Needless to say it was an experience that I will never forget, and one that has influenced my approach to medicine ever since.
The diagnosis was a surprise to everyone. There were of course red flags that appropriately led to the ordering of an MRI, but I really didn’t expect it to find anything. It wasn’t based on the fear of a lawsuit or the discomfort with uncertainty, influences that dictate the practice of medicine far too often, with the latter being considerably more of a problem (in my opinion). There was enough to support embarking on an encephalic expedition, but nothing so blatant as to really raise my suspicions. My guard was down and my emotional response to the situation was intense. Later, after care of the patient had been transferred to the pediatric oncology service, my introspective nature took over.
The sound of approaching hoofbeats
In medical school, there is an old adage often passed down to learners by seasoned physicians that serves to rein in an overly broad differential diagnosis, particularly when it includes increasingly unlikely etiologies. When one hears the sound of approaching hoofbeats, the inexperienced students are cautioned, one should expect to see horses rather than zebras. The world of medicine rarely mimics an episode of House, M.D.
Horses, though sometimes serious and even deadly, are common conditions much more likely to cause a patients symptoms than some rare and exotic entity. Most fevers are benign and self-limited viral infections rather than Ebola, for example. Most headaches are tension headaches and not neurocysticercosis. Thus, horses make the most sense in the setting of approaching hoofbeats. What is considered a horse or a zebra may change depending on your patient population or location, however. Measles is a horse in some areas of the world, but it is most certainly a zebra in the United States…for now. But the cranium of the poor child in my anecdote contained a zebra, a roughly 7 cm by 6 cm zebra nestled in the left hemisphere of her brain.
A point of divergence
When a medical practitioner emerges from such an emotional encounter, in particular when there is a bad outcome that they believe could have been prevented or ameliorated in some way by a different approach, they stand at a potential point of divergence in their timeline. How the events are internalized might result in far-reaching ripple effects that forever alter the way they practice medicine. Unfortunately, the worst way to internalize these encounters is the one most hardwired into the human brain.
I’ve seen this happen in real time with residents. And I have worked with experienced older pediatricians who years later will still relate the story of how they came to always (insert some particular test or treatment) in certain situations regardless of what the evidence has shown us. “That’s the art of medicine,” they express, failing to realize that the painting they’ve created is more George Bush than Georges Seurat.
Despite my bias against bias, it would be foolish of me to deny the plain and simple fact that in the practice of medicine our prior experiences inform our future choices. For medical students and residents, there is certainly ample opportunity to learn from mistakes made when caring for ill patients. This is a process that should never end, although the opportunities are hopefully fewer with increasing knowledge and experience.
Introspection and assimilation
The more challenging path requires introspection and the appropriate assimilation of the emotionally significant and uncommon occurrence into one’s practice. This newfound experience can then serve as a means of teaching others, of emphasizing historical red flags and abnormal physical exam findings that should lead to reasonable testing, and of improving communication with patients and families. This allows for growth as a physician and as a person rather than regression.
The easier route would be to lower the bar for ordering head imaging in patients with signs or symptoms that could be associated, even if unlikely, with serious pathology in the brain. To allow a few more black and white stripes to mingle with the chestnut and palomino in our imaginary herd. And it seems intuitive considering the horrendous effect on the patient and her family in this case. But the ultimate destination of this route is the placement of more importance on personal experience than on objective evidence, and it is a dangerous direction to take.
The availability heuristic
Prior experience affects future decisions. There isn’t any doubt about this. But it can be taken to an extreme in the practice of medicine in part because of what psychologists call the availability heuristic. You may be unfamiliar with the term heuristic, but you have experienced the phenomenon on a daily basis. Heuristics are simple rules that allow us to make decisions more efficiently by focusing on key variables of a complicated problem. They act to influence our decisions beneath conscious awareness and work pretty well for the most part. But because they can sacrifice accuracy for speed, they sometimes result in cognitive bias.
The availability heuristic is a specific example that is frequently employed by physicians. It warps our perception of the likelihood of a diagnosis because we tend to place more emphasis on that which can be easily brought to mind, and emotionally charged experiences are typically the ones we recall most readily. For the remainder of my career, whenever I care for a patient with a presentation remotely similar to the one mentioned at the beginning of this post, and even if the indications for obtaining neuroimaging aren’t present, there is a decent chance I will think of this child. In time I may lose my memory of her face or the faces of her parents, but I don’t think I will ever forget that empty feeling I felt in the pit of my stomach.
My human nature, developed through hundreds of thousands of years of evolution, may demand action. It may attempt to fool me into thinking that a zebra is more likely to be galloping towards me than it truly is. And our human nature is a powerful force. The key to fighting against it in situations like this is to be armed with an understanding of how good thinking can go wrong, and how the human brain is hardwired to make rash judgments.

   Changing Your Mind
Mark Crislip
Link
Why is my mind so clean and pure? Because I am always changing it.
In medical school the old saying is that half of everything you learn will not be true in 10 years, the problem being they do not tell which half.
In medicine, the approach is, one hopes, that data leads to an opinion. You have to be careful not to let opinion guide how you evaluate the data. It is difficult to do, and I tell myself that my ego is not invested my interpretation of the data. I am not wrong, I am giving the best interpretation I can at the time. For years I yammered on about how it made no sense to give a beta-lactam and a quinolone for sepsis until a retrospective study suggested benefit of the combination. Bummer. Now when I talk to the housestaff about sepsis, I have to add a caveat about combination therapy. It is why my motto is, only half jokingly, ”Frequently in error, never in doubt”.
At what point do you start to change you mind? Alter your message as a teacher? Have new behavior? Medicine is not all or nothing, black and white. Changes are incremental, and opinions change slowly, especially if results of a new study contradict commonly held conclusions from prior investigations.
Nevertheless, I am in the process of changing my mind, and it hurts. I feel like Mr. Gumby.
It is rare that there is one study that changes everything; medicine is not an Apple product. Occasionally that there is a landmark study that alters practice in such a dramatic way that there is a before and after. As I write this I cannot think of a recent example in infectious diseases, but I am sure there is one. The problem is that once practice changes, it seems as we have always done it that way.
For me, three is the magic number. One study that goes against received wisdom warrants an ‘interesting, but give me more.’
Two studies, especially if using different methodologies with the same results gives and well, two is interesting, but I can argue against it.’ However, with two studies the seed of doubt is planted, waiting to be watered with the water of further confirmation. Yeah. Bad metaphor.
Three studies with different methodologies independently confirming new concepts? Then I say, ‘I change my mind. My brain hurts.’
There are now three studies concerning the issue of efficacy of the flu vaccine in the elderly. You might remember my discussion of the Atlantic article several months ago. In that entry I discussed two articles that suggested the flu vaccine may be less effective in the elderly than the studies demonstrated.
The argument was that the elderly who received the influenza vaccine were healthier at baseline than those that didn’t receive the vaccine and the deaths during flu season was not due to the protection from the vaccine, but due to the fact that healthier people are less likely to die when they get ill. In part this was demonstrated by showing decreased deaths in vaccinated populations when influenza was not circulating. If insomnia is a problem, you can go back and read my post. To quote my favorite author, me, I said
“One, it is an outlier, and outliers need confirmation. The preponderance of all the literature suggests that influenza vaccine prevents disease and death. If you do not get flu, you cannot die from flu or flu related illnesses. When outliers are published, people read them, think, “huh, that’s interesting”, but there is going to have to be more than one contradictory study to change my practice. But if “study after study” shows mortality benefit, and one study does not, it is food for thought, but not necessarily the basis of changing practice. The results, above all, needs to be repeated by others… In medicine we tend to be conservative about changing practice unless there is a preponderance of data to suggest a change is reasonable. Except, of course, if our big pharma overlords take us to a good streak steak house.”
Now we have a third article, “Evidence of Bias in Studies of Influenza Vaccine Effectiveness in Elderly Patients” from the Journal of Infectious Diseases.
In the study they examined the records of the elderly in the Kaiser Health System, their vaccination records, and their risk of death. And the results were interesting.
“The percentage of the population that was vaccinated varied with age. After age 65, influenza vaccination increased until age 78 in women and age 81 in men, then decreased with increasing age. Vaccination coverage also varied in a curvilinear fashion with risk score, increasing with risk score to a risk score percentile of ∼80%, then decreasing. In addition, as the predicted probability of death increased, vaccination coverage increased. Vaccination coverage was highest among members with a probability of death of 3%–7.5%. Those with a predicted probability of death in the coming year of 17.5% had a decreasing likelihood of influenza vaccination”
They then looked at mortality when flu was not circulating.
“A change in the pattern of vaccination had a striking effect on mortality. For members > 75 years old who had been receiving influenza vaccinations in previous years, not receiving a seasonal influenza vaccination was strongly associated with mortality in the months ahead (Table 1). A person who had received an influenza vaccination every year in the previous 5 years had a more than double probability of death outside the influenza season if he or she missed a vaccination in the current year, compared with a person who was vaccinated as usual (odds ratio, 2.17; P < .001). On the other hand, if a person did not receive any seasonal influenza vaccination in the previous 5 years, then receipt of a vaccination in the current year was associated with a greater probability of death. “
If they had a history of flu vaccine for five years and missed it, the probability of death went up.
If they did not have a flu vaccine for five years and got one, the probability of death went up.
They suggest in the first case, the patients may have had an increase in their co-morbidities and as a result did not get the vaccine and died of underlying diseases. Their increased risk of death was from accumulating prior illnesses.
In the second case, people who were healthy and did not seek care subsequently developed diseases that lead them to a doctor who advised the vaccine. Their increase risk of death was due to new illnesses.
Either way, the uptake of the flu vaccine is more complicated than I had suspected and makes interpretation of efficacy of the vaccine in prior studies harder to evaluate. The table shows an unexpected relationship between age, risk of death and use of the flu vaccine.
They say in the discussion
“We showed that, despite strong efforts to increase vaccination among the elderly population, vaccination is relatively low in the oldest and sickest portions of the population. Persons 65 years old with a 17.5% chance of death in the upcoming year are less likely to receive the influenza vaccine. Because persons who are most likely to die are less likely to receive the vaccine, vaccination appears to be associated with a much lower chance of dying; thus, the “effectiveness” of the vaccine is in great part due to the selection of healthier individuals for vaccination, rather than due to true effectiveness of the vaccine. Previous studies have argued that worsening health is associated with increasing vaccination. We found this to be a curvilinear relationship, in which increasing illness means increasing vaccination, up to a point, and then, as people come closer to the end of life, there is a decrease in vaccination coverage.”
They do not say the vaccine is not effective, but they suggest that there is a bias that may make the vaccine appear more effective in the elderly than it really is. Reality is often more complex than one would think at the beginning.
After three studies I am reasonably convinced that efficacy of the flu vaccine in the elderly is potentially not as well understood as I had thought.
So do I think the flu vaccine is no longer useful in the elderly? No. I still think it is a reasonable intervention but it may not have the efficacy I would like. But I have always known that, for a variety of reasons, the flu vaccine is not a great vaccine. But it is better than no vaccine. There are, as discussed in the earlier post on the vaccine, many lines of evidence to show that the flu vaccine has benefit; at issue is the degree of the benefit. Perhaps what is needed is a better vaccine with adjuvants or multiple injections to get a better result in the elderly, who respond poorly to the vaccine. Or perhaps it will be better to focus on increasing vaccination in those who care for or have contact with the elderly. But when I talk to my patients and residents, when I get to part about flu vaccine efficacy, I will be a little more nuanced, use more qualifiers. I will tell them that the vaccine is like seat belts. It does not prevent all death and injury, but if you had a choice, would you not choose to use seat belts?
In the end the data has to change the way I think about medicine, not matter how much it hurts.
Compare and contrast that with the anti-vaxers who have the belief that vaccines cause autism. They look for data to support the pre-existing belief and ignore contrary data. Opinion does not follow from data.
The most representative statement of their approach is on the 14 studies website where they say ”“We gave this study our highest score because it appears to actually show that MMR contributes to higher autism rates.”
The key phrase in the whole site. Data that supports their position is good, data that does not is bad. What makes a study good is not its methodology or its rigor, or its reproducibility, or its biologic plausibility, but if it supports vaccines casing autism.
Dr. Wakefield, as has been noted over the last week, had his MMR/autism paper withdrawn from Lancet not for bad science, but for dishonest science. In medicine you can be wrong, but you cannot lie. If the results of medical papers were shown to be fabrications, such as the papers of Scott S. Reuben, no one the medical field would defend the results. Dr. Reuben, as you may remember, was found to have fabricated multiple studies on the treatment of pain. Nowhere can I find web sites defending his faked research. No suggestions it was due to a conspiracy of big pharma to hide the truth. No assertions that he is still a physician of great renown. He lied and is consigned to ignominy. Physicians who used his papers as a basis of practice no longer do so, or so I would hope.
The response to Dr. Reuban is in striking contrast to the defense of Dr Wakefield, where bad research combined with unethical behavior, results in
“It is our most sincere belief that Dr. Wakefield and parents of children with autism around the world are being subjected to a remarkable media campaign engineered by vaccine manufacturers reporting on the retraction of a paper published in The Lancet in 1998 by Dr. Wakefield and his colleagues.
The retraction from The Lancet was a response to a ruling from England’s General Medical Council, a kangaroo court where public health officials in the pocket of vaccine makers served as judge and jury. Dr. Wakefield strenuously denies all the findings of the GMC and plans a vigorous appeal.”
Opinions did not change when the Wakefield paper was demonstrated to be not just wrong but false, the researcher’s behavior unethical, and the study could not be reproduced using similar methodologies. Instead, the defense of Dr. Wakefield became, well, like a Jim Carrey shtick. The Mask defends retracted autism research. Fire Marshall Bill on the medical literature. Jenny and Jim’s defense does make more sense read as comic performance art. Andy Kaufmann would have been proud.
I wonder if the more grounded in fiction an opinion is, the harder it is to change, the more difficult it is to admit error. I have to admit I cannot wrap my head around the ability of people to deny reality. It is the old Groucho line come to life, “Who are you going to believe, science or your lying eyes?”
So I will, I hope, keep changing my mind as new information come in. It is what separates real health care providers from acupuncturists and homeopaths and naturopaths and anti-vaxers. It is what some truly great minds admit to doing. As one deeper thinker and better writer than I said, kind of,
“The other terror that scares us from self-trust is our consistency; a reverence for our past act or word, because the eyes of others have no other data for computing our orbit than our past acts, and we are loath to disappoint them.
But why should you keep your head over your shoulder? Why drag about this corpse of your memory, lest you contradict somewhat you have stated in this or that public place? Suppose you should contradict yourself; what then? It seems to be a rule of wisdom never to rely on your memory alone, scarcely even in acts of pure memory, but to bring the past for judgment into the thousand-eyed present, and live ever in a new day. In your metaphysics you have denied personality to the Deity: yet when the devout motions of the soul come, yield to them heart and life, though they should clothe God with shape and color. Leave your theory, as Joseph his coat in the hand of the harlot, and flee.
A foolish consistency is the hobgoblin of little minds, adored by little statesmen and philosophers and divines and anti-vaxers. With consistency a great soul has simply nothing to do. He may as well concern himself with his shadow on the wall. Speak what you think now in hard words, and to-morrow speak what to-morrow thinks in hard words again, though it contradict every thing you said to-day. — ‘Ah, so you shall be sure to be misunderstood.’ — Is it so bad, then, to be misunderstood?”

Anti-anti-vax: Getting to the gist
Scott Gavura on March 29, 2012
Link
I’m currently putting the finishing touches on a presentation for the The Ontario Public Health Convention next week, where I’ll be speaking, with occupational therapist Kim Hébert, about the anti-vaccine movement and social media (SM): how antivaccine advocates use it, and the challenges and opportunities for public health advocates. I’m pleased to see Seth Mnookin, author of The Panic Virus and someone whose work is likely known to many SBM readers, is one of the keynote speakers – his perspective will be valuable for the public health crowd which has traditionally relied on fairly static “key messages” for disseminating information on vaccine safety and effectiveness. The panel discussion of which I am a part will be an examination of challenges and opportunities presented to public health advocacy, and particularly vaccine advocacy, in a Web 2.o environment. What seems clear is that the old public health channels don’t cut it anymore: these methods are distant and insufficient to address the wide and rapid spread of misinformation in an era of social media. We all remember the anxiety over H1N1 just a few years ago – and judging by the poor uptake of the vaccine, it seemed the anti-vaccine movement had some success in propagating fear, uncertainty, and doubt. I’d almost forgotten about this chestnut from the Health Ranger himself:
In about two minutes, Mike neatly compiles most of the still-common anti-vaccine gambits in use. It’s not keyboard cat popular, but almost 200,000 views is likely greater than any pro-vaccine video produced. But is it effective? I looked to the medical literature for any current articles on the tactics of the antivaccine movement. In contrast to the wealth of information online, particularly here at SBM, there is a much smaller set of papers in print. A friend of the blog covered one recent paper back in January, Anna Kata’s Anti-vaccine activists, Web 2.0, and the postmodern paradigm – An overview of tactics and tropes used online by the anti-vaccination movement. Kata neatly categorizes the disingenuous tactics and messaging used by the antivaccine movement. Tactics include:
    • Skewering the science of vaccine safety and efficacy, while trying to create legitimacy for unfounded or discredited theories of harm.
    • Shifting the hypotheses and the villain, from MMR, to thimerosal, to other “toxins”, and more recently, “too many, too soon”.
    • Censoring criticism, whether it’s at Age of Autism, Mothering.com, or other antivaccine sites that delete comments or restrict access to their events.
    • Attacking the opposition, whether it’s our own David Gorski or lawsuits against Paul Offit or Amy Wallace.
And then there are the tropes, known to SBM readers as they’ve all been covered in some form before:
    • “I’m not anti-vaccine, I’m pro-safe vaccine”
    • “Vaccines are toxic!”
    • “Vaccines should be 100% safe”
    • “You can’t prove vaccines are safe”
    • “Vaccines didn’t save us”
    • “Vaccines are unnatural”
    • “Choosing between diseases and vaccine injuries”
    • “Galileo was persecuted too”
    • “Science was wrong before”
    • “So many people can’t all be wrong”
    • “Skeptics believe…”
    • “You’re in the pocket of Big Pharma”
    • “I don’t believe in coincidences”
    • “I’m an expert on my own child”
I’m sure there are more than a few missing, as the spin continues despite the overwhelming evidence that vaccines, on balance, are safe and highly effective. Kata points out that the anti-vaccine movement has been around as long as there are vaccines: It exists despite the evidence, so no amount of evidence will ever cause it to disappear. She summarizes with a list of some potential solutions to arresting the propagation of this misinformation, noting the potential opportunities for social media campaigns to “immunize” against misinformation.
And that’s where a second article picks up, a more detailed examination of “Web 2.0” as the newest battlefield for both the antivaccine movement and for public health advocates. From Betsch et al, it’s entitled Opportunities and challenges of Web 2.0 for vaccination decisions and was published earlier this year in the journal Vaccine. The paper is a summary of a conference on vaccination decisions held in 2011. As someone who discusses vaccines directly with patients, but also via the interwebs, I found the paper discussed some of my personal questions about vaccine advocacy:
    • What is the most effective way for me, as a health professional, to identify and address antivaccine sentiment among my patients?
    • What is the most effective way I can use social media to present science-based information in a way that maximizes its usefulness to the largest audience?
The paper focuses on the impact of impact of Web 2.0, a term which may sound trivial to those who already contribute on a blog, as you’re already in a 2.0 environment. Briefly, any technology that facilitates user-to-user interaction can be thought of as Web 2.0. Social media (e.g., Facebook, Twitter, etc.) is any 2.0 technology that facilitates sharing, adapting, and crowd-sourcing material. Compared to the static websites of several years ago, most of the internet is now squarely 2.0. The key feature relevant to vaccines and web 2.0 is the ability to add a personal narrative to data. That is, users can adapt, promote, support and personally contextualize internet content easily now – sharing this article to your own Facebook feed, perhaps prefaced with your own comment on its merits, is one click away.
The emerging consequence of social media as it relates to vaccination decisions is the ability to attach a personal narrative to vaccine information.
Vaccination decisions
Betsch summarizes the considerations that go into vaccination decisions, which include recommendations from health professionals, social norms, previous experiences with, and beliefs about, vaccines, trusted individual and organizations, and related background knowledge. The key drivers of the decision are believed to be those that have an effect on risk perception – that is, the perceived risks and benefits of vaccinating, and also of not vaccinating. Both the frequency or likelihood of an event, and the severity of that event, may be considered.
And here’s where it gets interesting with Web 2.0. As I’ve noted before, it takes only a few minutes of reading anti-vaccine information to shift perceptions of relative risk and benefit, and to lower vaccination intentions. And social media makes it that much easier – by effectively using the power of a simple narrative. And narrative is among the most potent tactics of the antivaccinationist.
The narrative
I take great pains as a health professional to give the proper context for my recommendations. If questioned about MMR vaccines and autism, I might provide a response like, “Numerous studies have demonstrated that there’s no relationship between the MMR vaccine and neurodevelopmental disorders”. While that may be accurate, it may not be most effective way to address the underlying uncertainty. The individual benefit of vaccination may be hard to see (though it’s easier when there are outbreaks to point to.) The harms, whether real (injection pain) or just attributed to vaccines, may be more tangible. The antivaccine response, while wrong, is far simpler: “MMR has been linked to autism. The vaccine is dangerous.” The ability to link a personal, emotional narrative to an antivaccine trope is facilitated by Web 2.0 technologies – it can personalize impersonal information. Emotional appeals can push the actual evidence into the background. The producers of the Greater Good movie used this approach to deliver their antivaccine message: Three narratives of harm attributed to vaccines, around which all of the antivaccine tropes described above were packaged and delivered.
Betsch points out how the narrative, or anecdote, is a powerful communication device. It’s easily understood, concrete, seemingly credible, and emotional. Whether or not they’re causally linked to the vaccine isn’t relevant – narratives increase the perception of harm and of the resultant risk of vaccination. While there have been some attempts to compile narratives of consequences of vaccine avoidance (What’s the Harm? comes to mind), on balance, negative personal narratives on vaccines seem far more prevalent.
The gist
So the challenge for public health advocates in a Web 2.0 environment is twofold: We must be accurate and effective. Accuracy may mean more complexity and nuance to our messaging. Plus we need to be able to rapidly respond to emerging concerns, ideally in a way that facilitates dissemination and sharing via social media to the same recipients of antivaccine messages. And here’s where we come to the third paper, one which explores theories of risk perception and decision-making: fuzzy-trace theory. It’s examined in detail in a 2011 publication by Valerie Reyna, Risk perception and communication in vaccination decisions: A fuzzy-trace theory approach. According to this theory, we integrate information in two distinct ways:
     1.  Verbatim memories: the precise details of the facts
     2.  Gist memories: the essential meaning of the facts
Based on this theory, gist memories are derived not from verbatim facts, but are developed in parallel with verbatim memories. Because they’re subjective, they incorporate existing biases and understandings. Reyna argues that gist memories are relevant to reasoning and decision-making – and that gist is in fact preferred when exact answers are not required. Importantly, meaning is at the core of gist memories. She argues that traditional public health messaging, designed to warn and persuade, doesn’t provide the necessary meaning to support the development of a pro-vaccine gist. From this perspective, anti-vaccine messages can create strong gist memories: they provide meaning to adverse events related to vaccinations – even where no true relationship exists. The narrative is coherent, even if it’s not rational. Positive vaccine communication needs to do the same, appealing to values, such as how vaccinating protects those that cannot be vaccinated. Emotional resonance counts, too – messages can be resonant while remaining accurate and distilling risk perceptions in ways that support positive gist interpretations.
Conclusion
Social media has changed the way we share information on the internet, presenting challenges and opportunities to public health advocates. The ability to attach a personal narrative to vaccine communication has the potential to shift risk perception about vaccines. While research on best practices continues to evolve, it seems safe to conclude the following: Make the gist of your pro-vaccine message easy to extract, and make it memorable.
References
Betsch, C., Brewer, N., Brocard, P., Davies, P., Gaissmaier, W., Haase, N., Leask, J., Renkewitz, F., Renner, B., Reyna, V., Rossmann, C., Sachse, K., Schachinger, A., Siegrist, M., & Stryk, M. (2012). Opportunities and challenges of Web 2.0 for vaccination decisions Vaccine DOI: 10.1016/j.vaccine.2012.02.025
Kata, A. (2011). Anti-vaccine activists, Web 2.0, and the postmodern paradigm – An overview of tactics and tropes used online by the anti-vaccination movement Vaccine DOI: 10.1016/j.vaccine.2011.11.112
Reyna, V. (2011). Risk perception and communication in vaccination decisions: A fuzzy-trace theory approach Vaccine DOI: 10.1016/j.vaccine.2011.11.070

When Doctors Refuse to Believe Evidence
Paul Offit’s new book covers the evidence for many surgeries, medications, and screening tests that have been proven ineffective and harmful yet are still being used by doctors who refuse to follow the science.
Harriet Hall on November 17, 2020
Link
Science-based medicine is all about testing medical ideas against reality. If there is abundant evidence from well-designed controlled clinical trials that a treatment is safe and effective, science-based doctors can feel justified in using it. If the evidence shows that the treatment doesn’t work and does harm rather than good, they abandon it. At least that’s what they logically should do. But all too often, that’s not what actually happens. Sometimes doctors refuse to believe the evidence and persist in doing things that don’t work and cause harm, including popular surgeries, medications, and screening tests. You might think that’s a rare occurrence; but it’s actually very common, as Paul Offit shows in his new book Overkill: When Modern Medicine Goes Too Far.
He covers 19 common beliefs that aren’t true, that have been repeatedly disproved by good scientific studies. He explains how and why the false beliefs arose and why they have persisted, describes the many studies that have produced good evidence that the beliefs are false, explains why some studies have produced different results, explains the science behind why a treatment works or doesn’t work, and ends by offering suggestions to overcome typical arguments for the persistence of false beliefs and suggestions about how to improve medical practice.
The false beliefs refuted
Clarification added 11/17: Some readers found this confusing. No, it’s not a list of false beliefs; it’s a list of Offit’s refutations.
     1.  Fever doesn’t need to be treated. (More about this later, and you can also review Clay’s post on the topic.)
     2.  Finishing a course of antibiotics is often unnecessary. (Actually, the length of a course of treatment is arbitrary, not evidence-based; and it makes more sense to stop the antibiotics when the symptoms have resolved. And it may help prevent drug resistance. Discussed by Scott!)
     3.  Antibiotic drops don’t treat pinkeye.
     4.  Vitamin D supplements are rarely necessary, and they don’t do most of the things people think they do. (See Scott’s post)
     5.  Supplemental antioxidants don’t lower the risk of cancer and heart disease; they increase it . (Another post by Scott)
     6.  Testosterone for “Low T” is dangerous and unnecessary. (See David’s post)
     7.  Baby aspirin doesn’t prevent first strokes or heart attacks and it can cause bleeding complications. It does work to prevent strokes or heart attacks in patients who have already had one. (Scott again)
     8.  If allergenic foods are introduced to infants at an early age, they will be less likely to develop allergies. (Discussed by Scott)
     9.  Reflux medicines for fussy babies should be avoided. Babies spit up; that’s normal. The drugs don’t reduce regurgitation or clinical symptoms, yet they are prescribed for one out of every ten babies in the US. (Discussed by Clay)
     10.  Sunscreens offer false security; users spend more time in the sun and get more skin cancers.
     11.  Prostate cancer screening programs do more harm than good. (Discussed by David)
     12.  Thyroid cancer screening programs don’t save lives. (David again)
     13.  Breast cancer screening programs aren’t exactly as advertised. For every 1000 women screened, 7 lives will be saved but 19 will be diagnosed with a cancer that never would have killed them, and of those 19, 99% will have surgery, 70% radiation, 70% hormone therapy, and 25% chemotherapy, all without benefit. (David)
     14.  Heart stents don’t prolong lives.
     15.  Surgery for knee arthritis is unnecessary, especially arthroscopic procedures. Even total knee replacement is problematic: 85% of patients had less pain but so did 65% of the nonsurgical comparison group.
     16.  Mercury dental fillings should not be removed. They’re not causing health problems, and removal increases mercury exposure.
     17.  Vitamin C doesn’t treat or prevent colds.
     18.  The standard advice to treat sprains with RICE (rest, ice, compression, and elevation) may do more harm than good: all four reduce blood flow and retard healing, as do anti-inflammatory drugs.
     19.  Teething doesn’t cause fever. (Clay)
For some of these items, you will say “I knew that.” Others may surprise you or strike you as wrong; but strong supporting evidence is provided in the book. We have covered several of these issues on SBM. You may not be ready to give up long-standing beliefs and practices. If you question some things on the list, I urge you to read the book and try to consider without bias the evidence Offit has marshalled.
The book has persuaded me to change my mind about some things. I had read arguments against treating fevers before, but I always rationalized that lowering body temperature wouldn’t make that much difference and that treating fever makes patients feel better. Offit points out that fever is a natural response that evolved in all animal species because it was beneficial; he says “We’re not supposed to feel better.” Fever is a signal that we should rest and let our body fight the infection at a higher temperature that allows every aspect of our immune system to work better. There is reassuring evidence that letting a fever from infection run its course will not “fry the brain” or kill the patient like heatstroke or malignant hyperthermia can, and that giving Tylenol before vaccination will not prevent a febrile seizure. So I have resolved that the next time I run a fever I will just go to bed and tough it out. It’s a good excuse for a mini-vacation. And if I sprain my ankle, I will not put ice on it, and I will try to tolerate mild activity.
Why?
If the evidence is so clear, why doesn’t everyone accept it? Offit gives several possible reasons.
    • Ignorance: no one can keep up with all the published studies, and some of the studies are poorly designed and report misleading results.
    • Fee for service: there are financial incentives for doing useless procedures. Solution: salaries that don’t depend on the number of procedures performed.
    • Patient satisfaction: patients want doctors to do something and are more satisfied when they get surgery or a prescription; they don’t have the knowledge to know what’s best for them and they have to depend on the expert (the doctor) for guidance.
    • Inertia: doctors want to continue doing what they have always done and what they have been taught to do. Their personal experience (which we know is not a reliable guide) has convinced them that it works, and it is painful for them to admit that they have actually been harming patients all those years.
    • Medical marketing: health advertising is a multibillion dollar industry that is poorly regulated. The U.S. and New Zealand are the only two countries that allow direct-to-consumer advertising for pharmaceuticals.
    • Fear of lawsuits: A jury awarded $986,000 to an alleged psychic who believed the MRI ordered by her neurologist had destroyed her psychic powers. Doctors may be sued for neglect if they refuse to take actions that other doctors take. Offit’s solution: eliminate jury trials; let a panel of medical experts decide if malpractice has occurred.
The take-home message: Trust science
This is an important book that is well written, accessible to laymen, and provides extensive evidence (51 pages of notes and references). Medicine would be healthier if all doctors and patients would read this book and follow the science.

Ignorance is Strength?
Findings from a recent consultation suggest that consumers don’t want health claims to be supported by evidence. Do consumers really prefer ignorance over evidence? Or is this the product of a industry campaign to derail new, science-based regulations?
Scott Gavura on April 6, 2017
Link


Back in September, I wrote about the consultation undertaken by Health Canada (Canada’s version of the FDA) to overhaul the way it regulates dietary supplements (they call them, somewhat misleadingly, “natural health products”.) Health Canada is proposing a move towards a risk-based licensing framework that will apply to all types of self-care products. This will eliminate the special consideration currently applied to natural health products which undergo only a cursory review, yet are permitted to make scientifically-unsubstantiated claims on their labels. Health Canada’s newly-proposed regulatory approach will more tightly link specific efficacy claims with the requirement to provide scientific evidence to justify these statements. Low-risk products, like the pseudoscience of homeopathy, wouldn’t be permitted to have scientific claims at all In fact, they wouldn’t even be reviewed by Health Canada, eliminating the perception among Canadians that Health Canada’s approval is a signal that a natural health product is both safe and effective.
The proposal, which I noted wasn’t perfect, is much more aligned with science than the current licensing standard, which are, for lack of a better word, myth-based. Health Canada will currently approve virtually any product, with seemingly any claim, and for which zero credible evidence needs to be provided. While health regulations should exist to protect consumers, Health Canada has consistently given manufacturers the upper hand, prioritizing a company’s desire to sell a product over a consumer’s right to a properly regulated marketplace with safe, effective, and accurately-labelled products. CBC Marketplace proved this when it obtained approval for a fake homeopathic remedy, substantiated by a few photocopied pages from a homeopathy textbook. After that embarrassment, Health Canada tweaked its regulations, but not enough to really address the root cause of the problem, which continues to be a regulatory double standard. Being based in evidence, I predicted that the supplement industry wouldn’t be happy with the new proposal. And judging by the feedback to Health Canada’s consultation, it looks like an effective campaign was mobilized. Because if these results truly represent the views of Canadian consumers and health professionals, neither group seems to care about the accuracy of health claims made by supplement manufacturers – they actually oppose proposal to make them factual. Do consumers and health professionals really prefer ignorance over facts? 
The consultation
The consultation was launched in September and closed in October 2016, allowing anyone to contribute to discussions about how different types of self-care products (including cosmetics, dietary supplements, and non-prescription drugs) are regulated in Canada. The key components of what Health Canada is proposing are as follows:
    • Products with similar risk profiles would be treated (from a regulatory perspective) in the same way.
        ◦ This will ensure consistent and easier-to-understand rules for products whether they are supplements or drug products.
    • Health Canada would review claims based on a new definition of a health claim.
        ◦ Companies would be required to submit scientific evidence to support claims.
    • A risk-based approach would be implemented which would allow Health Canada to withdraw unsafe products.
Participants were invited to review a policy document before commenting. However it appears large number simply pasted in identical responses, in some cases seemingly not even responding to the facts of the proposal itself.
The response
The full report on the responses is available on Health Canada’s website. In total there were 3,567 responses of which 2,588 self-identified as consumers:
Health Canada noted that responses were “polarized” and “diverse”. There was mixed- to-negative support for grouping and regulating products based on risk (versus the status quo, which is a lowered bar for natural health products). Most respondents suggested that this approach would give them less, not more, confidence in decision-making:
It’s the next question where I wondered if the report was written by George Orwell. When it comes to scientific data, consumers and health professionals apparently don’t want it forming the basis for health claims:
The consultation notes that there were fears of this requirement affecting the “affordability, availability and diversity” of products sold, which suggests that many didn’t understand the proposal itself, which was related to claims, not availability.
I can’t think of another consumer good or any marketplace of any kind where consumers would actively support allowing manufacturers to make unsubstantiated claims. Holding manufacturers to account harms no-one except the manufacturer’s financial interest. Consumers would benefit from access to only substantiated claims, rather than the current standard of unsubstantiated claims. Yet overwhelmingly, this aspect of the consultation was not favored by respondents.
The puzzling responses continued in question 8, which asked if a unique product identifier should be placed on the package to support identifying problematic products, or allowing easy identification in the case of a recall. Again, there was more opposition than support for this proposal. Why this would be opposed at all is a bit baffling to me, as it harms no-one and only has the potential to benefit consumers. At least healthcare professionals were slightly in favour:
Overall, respondents were not supportive of the proposed strengthened regulations, and seemed concerned that it will limit access to products:
Industry mobilization campaigns
The report comments on two mobilization campaigns:
Health Canada is aware of two public mobilization campaigns which were undertaken during the online consultation period.
First, a petition was developed and publicized that urged the Prime Minister and Cabinet Ministers to protect the interests of local producers of medicinal herbs against limitations that might be imposed by a regulatory requirement for proof of efficacy. This petition received 1,514 signatures.
A second campaign, which implied that Health Canada is proposing to regulate NHPs as prescription drugs, encouraged members and supporters to complete the consultation questionnaire using elements from a template of answers which was created and published online. The group generally opposed all aspects of the proposed modernization and encouraged members and supporters who shared this view to respond to the consultation and use its template when doing so. The template provided clearly affected the overall tone and content of responses to the consultation questionnaire. In fact, of the 3,300 questionnaires completed by participants, 493 (or 15%) responded to six or seven of the seven closed-ended consultation questions exactly as the advocating group had proposed. An analysis of the open-ended responses provided by these participants also showed a close match to the text content suggested by the advocating group, which suggests that respondents may have taken their lead from the campaign.
The impact of this campaign on the overall results of the consultation cannot be discounted. The results of the campaign are important overall context for understanding the consultation results, as these respondents are typically exclusively interested in the future of NHPs under the proposed modernization rather than in the future of all self-care products. It is important to note that presently, when looking at cosmetics, NHPs and NPDs, the most modern set of regulations and the most risk-based regulatory structure is the one in place for NHPs. As such, it is possible that the potential benefits the proposed approach would bring are less evident for NHPs specifically when compared to cosmetics and NPDs.
What happens next?
As Carly Weeks noted in The Globe and Mail, Health Canada, to its credit, seems undeterred:
On Friday, the department released a report on the consultation process that revealed strong opposition to changing the way natural health products are regulated. Much of the opposition came from members of the public responding to an advocacy campaign led by members of Canada’s natural health products industry, which warns that the new system would limit the availability of vitamins, supplements and similar products.
And:
While the government is considering all the responses, it remains committed to updating the regulatory system, Ms. Bombardier said. According to the report, some of the public opposition stems from misconceptions promoted by the natural health industry, such as the inaccurate assertion that Health Canada wants to regulate natural health products the same way it regulates prescription drugs.For instance, the Canadian Health Food Association, which led a campaign against the proposed changes, suggests on its website that the new system will reduce oversight on some natural health products and increase it on others, requiring “a level of evidence similar to what is needed for drugs.”
Health Canada has announced another round of in-person and online consultations, starting this month. If the feedback to the initial survey is any sign, we should expect the supplement industry to continue to mobilize and push hard to derail any attempts to strengthen regulations. Ultimately, Health Canada is accountable to all Canadians, not just supplement manufacturers. Health Canada’s new proposals have the potential to repair much of the damage that’s resulted from a lax approach to regulation. Restricting manufacturers from making claims that aren’t backed by good science isn’t a question that should even need to be asked – nor should it be subject to polling. It’s an essential component to fairness in any health market that prioritizes public health over manufacturer interests.

Why People Continue to use SCAMs
Mark Crislip on July 25, 2014
Link
I remain curious as to why people use, and continue to use, useless pseudo-medicines. I read the literature, but I find the papers unsatisfactory. They seem incomplete, and I suspect there are as many reasons people choose a pseudo-medicine as those use them.
There are numerous surveys on what SCAMs people use. Designing and offering these surveys to every possible medical condition is a growth industry: the old, the young, cancer patients, AIDS patients. All need be asked which SCAM they use. It seems to be a ready way to get a quick entry in your CV, but which SCAM is used does not speak to the why a particular SCAM is being used. Why try acupunctures, say, instead of reflexology?
There are numerous reasons suggested for why people partake of SCAMS as a general concept: dissatisfaction with standard medical care is a common one but is not always supported in the literature. Gullibility, ignorance, and stupidity are often credited, none of them are particularly valid. Dr. Novella covered the topic in 2012. There is some data to suggest that which SCAM and why is a moving target, changing over time.
The problem with surveys is that the answer given depends in part on how the question is phrased. And as I get older the more I suspect that free will and conscious, rational thought are an illusion at worst and rare phenomena at best. Sure, I can make trivial choices, deciding to hit a 7 or 6 iron on number 5 at Rose City, depending on wind and tee location. That is, I hope, a rational decision based on current conditions. And I can still snap hook the ball into water that should never seriously come into play.
But so much of what I think burbles up from some subconscious mental process. After I come to a decision or have an opinion, I devise an after-the-fact conscious justification for my position. I hear myself making up the reasons on the fly, half-wondering who made the decision in the first place.
This is most acutely true in ID and happens more and more as my career progresses. As I hear a case presentation from the resident the probable diagnosis pops into my head. Since I have to teach the resident why I think it is a particular diagnosis, I explain my reasoning, but I am often aware that it is all after-the-fact hand waving. I do not really know why I thought it was a liver abscess that is causing the fever.
Just where these ideas come from is unknown, and a little creepy since it is not the conscious me who is doing the work. It is part of the reason why, along with free will, I suspect consciousness is a minor and unimportant aspect of the human condition. So much seems to go on in my brain over which I have no control.
Most of the brain’s work is done while the brain’s owner is ostensibly thinking about something else, so sometimes you have to deliberately find something else to think and talk about.~ Neal Stephenson, Cryptonomicon
Declared motivations for behavior are just one big post hoc ergo propter hoc fallacy. So when people offer explanations for why they participate in a particular SCAM, I am skeptical. I suspect it is all an after-the-fact rationalization. People’s motivations are black boxes and I suspect most do not have a good understanding as to why they use SCAM or do anything else.
Still, it doesn’t stop one from making broad generalizations. I find it interesting how various biases lead to erroneous conclusions about the way the world works. As I have mentioned before, our brains have evolved to survive reality, not to understand it.
In part what separates those who subscribe to the notion of science-based medicine and those who practice pseudo-medicine are explicit criteria for accepting evidence of therapeutic efficacy combined with an understanding of all the logical fallacies to which we are prone.
The most compelling evidence are randomized, placebo controlled, double-blind studies. The least compelling is personal experience and testimonial. I do not consider making stuff up as evidence, a standard not always shared. For most people the order is reversed, and so often it is the story of the friend’s cousin who had their disease cured with some pseudo-medicine, and what do you have to say to THAT, Mr. Smarty Pants Skeptic?
I take Feynmans quote to heart:
The first principle is that you must not fool yourself – and you are the easiest person to fool.
Those who practice pseudo-medicine have to ignore it. And there are many ways in which we can be fooled in to continuing a pseudo-medicine after, for whatever reason, we have decided to give it a try.
It has been suggested that illusions of causality (are) at the heart of pseudoscience including pseudo-medical interventions. After reviewing
how superstitious beliefs of all types are still happily alive and promoted in our Western societies
the authors state, or understate,
it is not easy to counteract the power and credibility of pseudoscience.
One cognitive bias is that with people will credit control of events even though they are not responsible for the event. The classic examples are gamblers and athletes with their useless rituals. Interestingly, the more a person is involved in an activity, the stronger their illusion of control. Health and illness tend to provide an opportunity for deep involvement. Few issues are as important as personal health and offer more risk for being fooled.
…people (and, arguably, other animals as well) trying to obtain a desired outcome that occurs independently of their behavior tend to believe that they are causing the outcome.
And people often credit causality of two events even when there is no credible connection between the events. We love to find causality where none exists.
Medical conditions are particularly prone to cognitive errors. Diseases and their symptoms wax and wane spontaneously and more severe illness tends to lessen overtime (regression to the mean). People usually seek care when symptoms are at their worst and so will get better no matter the intervention, effective or not. However, the bias will be to credit the intervention for the resolution.
I see this all the time in my practice. A patient with no good diagnosis for a treatable infection gets better on antibiotics. Is it that they got better and were on antibiotics (true-true and unrelated is the medical shorthand) or they got better because they were on antibiotics. I tend towards the former but it is very hard to convince doctors and patients that the latter is not true. I wonder how much inappropriate antibiotic prescribing is due to illusions of causality rather than pleasing the patient and time constraints, which are the usual explanations.
The hypothesis, ‘the illusion of efficacy of an intervention is an important factor regarding why people believe worthless therapies are effective,’ was studied. In a computer simulation:
Participants were asked to imagine being a medical doctor who was using a new medicine, Batatrim (i.e., target cause), which might cure painful crises produced by a fictitious disease called Lindsay Syndrome. Then, participants were exposed to the records of 100 fictitious patients suffering from these crises, one per trial. In each trial, participants saw three panels. In the upper one, participants were told whether the patient had taken the medicine (cause present or absent). In the second panel, participants were asked whether they believed that the patient would feel better. Responses to this question were given by clicking on one of two buttons, ‘Yes’ or ‘No’. The purpose of this question was to keep participants’ attention. The third and lower panel of each trial appeared immediately after participants gave their response. It showed whether the fictitious patient was feeling better (i.e., effect present or absent). In Group High p(C), 80 out of the 100 patients had followed the treatment and 20 had not. In Group Low p(C), 20 patients had followed the treatment and 80 had not. In both cases, 80% of the patients who took the medicine, and 80% of those who did not, reported feeling better.
So no matter the intervention, patients improved 80% of the time.
There were then asked
To what extent do you think that Batatrim is the cause of the healings from the crises in the patients you have seen?’ (Causal question), and ‘To what extent do you think that Batatrim has been effective in healing the crises of the patients you have seen?’ (Effectiveness question)
Those who were in the Group High were more likely to judge the Batatrim effective, which is probably how people assess therapies in real life, even though the results were random.
Interestingly, they were less likely to rate Batatrim as causing the improvement, even in the Group High. Thinking in terms of causality is not how people assess therapies in real life.
So they suggest that thinking in causal rather than in effectiveness terms, may help decrease the illusion of efficacy. Understanding potential causality, prior plausibility, is difficult. Most people have no reason to understand the biomedical sciences in the depth that would allow them to understand why homeopathy or acupuncture is a crock. So when they take useless nostrums and improve they probably think in terms of efficacy of the treatment rather than causality and credit the useless nostrum.
Not really a surprise as a result, except for the potential beneficial effect of getting people to think causally about an intervention. When a person knows they improved after, say, acupuncture, they always look at you (me) like you are dense when you suggest they did not get better from the acupuncture. Of course they got better from the intervention. Duh.
Quite possibly, the effectiveness question is the one that is most frequently used by lay people when inquiring about the efficacy of medical treatments, and pseudoscientific claims in general, but it should be noted that it is probably a misleading question. Stating that a treatment is not effective when all the people who we know that have followed it feel better makes no much sense. However, if we were asked whether the treatment is the real cause of the recovery of those patients, we would have to look for alternative potential causes (even hidden causes). This process forces participants to consider all the evidence available, something that they do not always do and that other questions may not require.
This study, even as a simulation, helps add to the understanding not only as to why people continue to use a given useless therapy, but perhaps one way to counter their use, although the authors do note that it is easier said than done:
Although a good knowledge of scientific methods is always desirable, one problem of such a strategy is that it requires, first, to convince people that science is something they should pursue (something quite difficult in pseudoscience circles of influence), and second, perhaps even more difficult, to convince people to use control conditions, to reduce the frequency with which they attempt to obtain the desired event, so that they can learn that it occurs equally often when they do nothing.
But wait. There’s more. The use of pseudo-medicines is more nuanced than illusion of causality.
I expect therapies to do something, not only the primary effect, but to have side effects. It turns out that doing nothing is an important factor in people thinking a useless therapy is effective.
In PLOS One was the recent article “The Lack of Side Effects of an Ineffective Treatment Facilitates the Development of a Belief in Its Effectiveness.”
They suggest:
While most would agree that people frequently resort to those treatments they believe are more effective, we propose that the reverse also holds: frequent use of a treatment, because of the lack of side effects or other considerations, fuels the belief that it is effective, even when it is not.
They use the ultimate in nothing, homeopathy, as an example, noting that use of the nostrum results in the causal illusion, which is increased when the process being treated has a high spontaneous resolution rate.
They note:
Basic research suggests that the more often a patient takes a completely useless medicine, the more likely she will develop a belief in its effectiveness. This is particularly true when the desired outcome (the healing) takes place frequently.
No one had looked at the consequences of medication side effects have in determining the belief that a useless therapy has efficacy. Their
prediction is that, because a lack of side effects encourages the use of the treatment with high probability, it facilitates the illusory belief that the treatment is working.
The used a computer simulation, a variation of the prior study. Students were asked to again treat a dangerous disease called “Lindsay Syndrome” with a drug called Batatrim. I am starting to think we should call all useless drugs Batatrim, but it would be too obscure a meme. Participants were divided into two groups:
The high-cost group was informed that Batatrim would produce a severe and permanent skin rash as a side effect in every patient who takes it.
and a low cost group whose patients had no complications from the Batatrim. Subjects were then shown the records of 50 consecutive patients with the disease and asked as to whether they would treat.
Whether or not the patient got better was determined randomly, but 70% of the time the fictitious patient improved.
After the treatment was decided and the computer randomly assigned the results, the subjects had feedback:
In the no-cost group, the outcome was displayed as a picture of a healthy face and the message, “The patient has recovered from the crisis”, whereas the outcome absence was displayed as a picture of an ill face (greenish, covered in sweat) identical to the one presented in the top panel of the computer screen, and the statement, “The patient has not recovered from the crisis.” … By contrast, the high-cost group was shown pictures and messages conveying not only the disease outcome, but also the side effects of Batatrim when it was used. Thus, whenever the medicine was given, the picture of the patient showed a skin rash, and the statement also included the words “…and has severe side effects.” Likewise, whenever the medicine was not given, the words “…and has no side effects” were added to the message.
At the end the subjects were asked to rate the perceived effectiveness of Batatrim. When there were no side effects reported subjects were much more likely to give the medication AND were much more likely to rate the drug as effective.
In this study, we have shown that knowing a medicine produces side effects prevented the overestimation of its effectiveness that is typically observed when the percentage of spontaneous remissions is high. We demonstrated that the mechanism by which this effect works rests on the lower frequency of the treatment usage exhibited by those participants who were aware of the medicine’s side effects.
To my mind, having no side effects is equal to saying that the therapy has no effects. But in the world of pseudo-medicines, having no effects and no side effects work together to fool the patient, by way of the causal illusion, that there is efficacy.
Having no side effects promotes use of the medication and for a process that has a high spontaneous resolution rate, promotes reinforcement of the causal illusion. Since the therapy is harmless, you use it more often and see it ‘work’ more often.
The solution? Perhaps it is actually education and awareness of how to think, at least in adolescents. For many adults it is probably too late.
We found that training a group of adolescents in the rational manner of making inferences about cause-outcome relationships decreased their illusory perceptions of causality in a subsequent non-contingent situation. Moreover, including a control condition in the positive contingency scenario allowed us to conclude that the lower causal ratings observed in the experimental group could not be solely explained by a general increase in suspicion in this group. Rather, the group specifically made more realistic judgments in the null contingency condition while preserving an accurate view of the positive contingency condition.
Whether this kind of educational interaction will result in long-term changes after the initial interaction is unknown and I would be skeptical. I suspect that rational thought is not the default cognitive mode. This is certainly the case for me. I always have to will myself to think rationally about topics. The causal illusion is powerful especially when combined with all the other cognitive biases simultaneously in action and the apparently natural resistance to changing one’s mind even as realty changes around you.
A foolish consistency is the hobgoblin of little minds, adored by little statesmen and philosophers and divines. With consistency a great soul has simply nothing to do. He may as well concern himself with his shadow on the wall. Speak what you think now in hard words, and to-morrow speak what to-morrow thinks in hard words again, though it contradict every thing you said to-day.
Still, it gives one hope, and that despite my lobbying to make Sysiphus the logo for SfSBM, the concept at the heart of the science-based medicine blog, accurate information, is fundamentally the correct one.

Why Do Things That Are Unlikely to Harm Us Get the Most Attention?
We are very bad at assessing risk, often giving the most attention to the things that are least likely to harm us. Geoffrey Kabat’s new book teaches us how to think more clearly about scientific studies of environmental health risks.
Harriet Hall on February 28, 2017
Link

Sharks get a lot of bad press and inspire a lot of fear, but in 2014 no one was killed by a shark in the US, while 36 people were killed by dogs, and 83 were killed by other mammals (including horses and cows). And tobacco kills a whopping 5.4 million people around the world every year.
Why do things that are unlikely to harm us get the most attention? In his new book Getting Risk Right, Geoffrey Kabat asks and answers that question. He says:
…we have been encouraged to worry about deadly toxins in baby bottles, food, and cosmetics; carcinogenic radiation from power lines and cell phones; and harm from vaccines and genetically modified foods… When looked at even the least bit critically, many of the scares that get high-profile attention turn out to be based on weak or erroneous findings that were hardly ready for prime time.
Cognitive pitfalls
Kabat explains the cognitive processes that underlie the paradox. The scary studies are often flawed and biased, and authors frequently make leaps way beyond the actual data; but we block out those considerations because of the visceral impact of scary findings. They “sound” true because of a broader cultural narrative. Positive studies impress us more than negative studies.
He discusses the illusion of validity and the value of negative thinking (thinking about what data we might have overlooked), how a weak hypothesis can take on a life of its own due to social and psychological factors, how a scientific consensus of no risk can be outweighed in the public mind by a few vocal activists, the difference between correlation and causation, Hill’s criteria of causation, bad science and voodoo science, the complex relationships between environmental pollutants and clinical variables and the danger of focusing on one risk at a time, the fact that studies showing associations with cancer can be found for nearly every food in your cookbooks, why most research findings are false, poor reporting by the media, publication bias, the importance of dose and timing, slow thinking vs. fast thinking processes, availability cascades, and many, many other factors that may mislead us.
He makes this important distinction between “hazard” and “risk”: the drain cleaner under your sink is a hazard, but it only becomes a risk if you drink it or get it on your hands. He points out the danger of believing one’s own hypothesis. He shows that invoking the precautionary principle may not be wise.
Threats that are invisible and not under our control (like ionizing radiation) evoke a much stronger reaction than things like smoking and obesity, which have been “domesticated” and may seem to be under our control.
The dangers of focusing on unlikely harms
When people are desensitized by the latest threats, they pay less attention to known serious health risks such as smoking. When a topic is popularized by the media, scientists are more likely to keep studying it even when the results are negative. Research money is diverted to that topic and is less available for topics that are more likely to be productive. Biases and agendas have undue influence. The confusion of conflicting studies erodes public trust in science. We need to understand the interactions between researchers, the media, regulatory agencies, and advocacy groups.
Successful research vs. problematic research
Kabat asks:
…what does successful research in the area of health and health risks look like and how does it differ from the research that draws our attention to sensational but poorly supported or ambiguous findings that never seem to get confirmed but have great potential to inspire fear?
To illustrate the difference, he examines two examples of successful research (kidney damage from Aristolochia and cancer due to HPV) and two examples of the other kind (cell phones and endocrine disruptors). He says:
If we have in mind a model of what a true advance in the area of public health looks like, this might provide a much-needed reference point for judging the many sensationalized findings that get so much attention.
Aristolochia: an herb that caused a mysterious disease and a worldwide public health threat
Kabat tells a fascinating detective story. Doctors in the Netherlands were seeing an epidemic of kidney disease, and they traced it to a mixture of Chinese herbs used at a local weight loss clinic, eventually determining that a toxic herb, Aristolochia, had been inadvertently substituted for one of the intended herbs. The pathology was reminiscent of that seen in a mysterious disease that spottily affected certain villages in the Balkans, and they eventually figured out that Balkan endemic nephropathy was caused by contamination of wheat supplies with a weed, Aristolochia. They found that aristolochic acid modifies DNA and causes a unique mutational signature, leading to kidney cancer many years after exposure. Also known as birthwort, Aristolochia is a widely used herbal remedy in China. Taiwan already has the highest rate of kidney disease and kidney cancer in the world, and it is estimated that 100,000 people have been exposed to the toxin and are at risk of developing kidney cancer down the road. More than 100 products containing Aristolochia can be purchased on the Internet. This is a cautionary tale: delayed toxic effects of herbs were missed by herbalists for centuries. So much for “natural” and “ancient wisdom!” Of the 51,000 new dietary supplements introduced since 1994 under DSHEA, only 0.3% have any documentation of their safety. There may be other Aristolochias waiting to be discovered.
HPV and cancer
Science was slow to recognize that viruses could cause cancer. The role of human papilloma virus (HPV) in cervical cancer was hard to detect, since the pathology progresses gradually over 15-20 years, and since many infections resolve on their own. We now have strong evidence that without HPV there would be no cervical cancer, and we have an effective vaccine that also protects against oropharyngeal cancers in both men and women. There is even the prospect that we might be able to virtually eradicate these cancers. The HPV story is a model for how basic science, molecular medicine, and technology can be effectively applied to real-world problems.
Cell phones and brain tumors
In 1993, David Reynard sued a mobile phone manufacturer for causing his wife’s fatal brain cancer, which happened to be on the side where she held her phone. Cell phones were a new technology that was not well understood, and this dramatic anecdote had enormous impact. Kabat describes the series of scientific studies that followed, and how the positive studies were plagued by bias and error. The best studies showed no association of cell phone use with brain tumors; in fact, regular use of cell phones was associated with reduced risk of glioma and meningioma in one study. And despite widespread cell phone use, the incidence of brain tumors has not risen. While long-term effects can’t entirely be ruled out, evidence from many avenues of research converges to suggest that cell phone radiation is not carcinogenic.
Endocrine disruptors
BPA (Bisphenol A) has been banned from baby bottles; does it really pose a threat? Kabat provides the best, most thorough analysis I have seen of BPA and other so-called “endocrine disruptors.” He puts the evidence into perspective. The alarm was raised when the British Medical Journal reported a decline in sperm counts and speculated that it might reflect a decline in male fertility and contribute to testicular cancer and other abnormalities. Researchers tried to tie the phenomenon to environmental exposure to chemicals that disrupt endocrine function. It is exceedingly difficult to specify a causal factor when dealing with low- and very low-level exposures in food, air, and water. Levels of “estrogen equivalents” from pesticides are on the order of one-thousandth the levels found in red wine or beans, and one-ten thousandth that found in cabbage. The estrogenic effects of BPA are 37,000 times weaker than estradiol.
Research into environmental exposures is problematic:
     1.  The effects may be subtle, transient, or nonexistent. Just because we can measure miniscule amounts in the blood doesn’t mean it is having a detectable effect.
     2.  Lifestyle behaviors such as smoking, drinking, obesity, etc. may overwhelm any effects of environmental exposures.
     3.  Genetic makeup may influence the ability to metabolize and detoxify these exposures.
     4.  Exposures typically involve mixtures rather than single substances, and exposures change over time and are difficult to measure.
     5.  If the effect is subtle or confined to a subgroup with particular vulnerability, few studies will have the ability to pick this up.
The BMJ researchers used poor methodology: they compared data from different countries at different points in time. Decline in sperm quality turned out not to be widespread, and in some locales, the quality actually increased. Sperm quality is known to vary with time since last ejaculation, scrotal temperature, prolonged sitting, season of the year, smoking, and drug use; these factors were not controlled for. The high levels of BPA reported in some studies may represent contamination of blood samples by BPA in medical devices in the hospital or clinic setting. Recent studies have shown that actual exposures are very low, and more than 99% of ingested BPA is efficiently metabolized.
Kabat suspects that scientists who favor the endocrine disruptor hypothesis have become too invested in their beliefs to do unbiased research. Enough research has been done to reassure most scientists that if there are any adverse effects, they must be very small in magnitude. Enough attention has been given to BPA and endocrine disruption, and it is time to direct research elsewhere.
Conclusion: Much valuable information in a small package
Kabat has packed a wealth of information into his 180 pages of text, and everything he says is copiously supported by references. There is much to learn here, not only about the four subjects covered in depth, but about how science works and the factors that can lead us astray. This book will provide you with defensive armor against alarmist headlines and it will help you judge the credibility of new studies. Highly recommended.

   Epiphany
Mark Crislip
Link
The Institute of Medicine report is a frequent ‘rebuttal’ to science based/real medicine. The argument is usually phrased something to the effect that since medicine can be dangerous, SCAM’s are legitimate. Of course, one does not follow the other. It is the equivalent of saying since you are old, bald and pudgy, I am young, have a full head of hair, and are thin. If every doctor and hospital were to vanish tomorrow like an episode of the Outer Limits, SCAM’s would be just a ineffective.
Despite the flawed logic of the comparison, I have always had an affinity for the estimates that 44,000 to 98,000 were (note the deliberate use of the past tense) killed each year in hospitals. There may be methodological flaws in the estimate but the ballpark figure is probably correct.
In 1999 there were 5000 hospitals in the US. Just one death a quarter would bring the number of deaths to 20,000, and one death a quarter is not that many deaths. Lest I sound hardhearted, everyone dies, 2.5 million a year, and often death occurs in the hospital. Against the background of the mortality of existence, a few ‘extra’ deaths would be lost in the background.
For an individual doctor, it would be indistinguishable against the background death rate in the hospital. What makes it even more difficult to track and recognize excess mortality is that each death may be due to a different breakdown in medical practice.
Amongst my many jobs is Infection Control. For twenty years I have chaired Infection Control for both the Legacy Health, a collection of 5 hospitals in the Portland-Vancouver area, as well as for Portland Adventist Medical Center. As Chair I get the joy of sitting on many other committees such as Quality Council and Pharmacy. I know all the way hospitals could kill and the endless efforts to try and improve and perfect medical practice to avoid these complications.
In 20 years of investigating outbreaks, hospital acquired infections, and deaths, I have yet to see two infection related deaths that are due to same cause. Every infection was reviewed and evaluated as a potential for improvement, and I think we practiced the best medicine we knew of at the time.
With one exception, the universal horse shit compliance with hand washing that was the norm 20 years ago. It always boggled my mind that it was difficult back in the day to get people to wash their hands. The information on efficacy was only 150 years old, after all. But otherwise we practiced state of the art medicine. With the perfect vision of hindsight, I can see that state of the art left much to desired. We didn’t have the studies to guide practice that we have now, and I anticipate that 20 years from now I will be rolling my eyes at how we practiced in 2010. I will sound just like Bones McCoy wandering throughout a 1980′s San Francisco hospital grumping about the butchers of the past and hoping I do not run into T.J. Hooker.
Unlike the hodgepodge of practices that comprise SCAM, medicine changes and mostly for the better. Change is always slow, and always painful, and more difficult to implement than one could ever anticipate, but if you read the medical literature, you have to change.
Hospital based medicine is mind bogglingly complex and difficult, and humans are limited in their ability to always function perfectly. The Institute of Medicine knew what it was doing when they entitled their report “To Err is Human.” And all too often we were not able to pinpoint a breakdown that lead to a complication or death
When I started practice back in the last century, I would have thought that hospital acquired infections were part of the price of taking care of ill and compromised patients. Sure, we can minimize infections, but wound infections, ventilator pneumonias, and line infections are going to happen. You can’t do the things we do to people and NOT get an infection.
Right?
Wrong.
What both administrations at my hospital systems have in common is a commitment to patient safety and over the last decade they have committed considerable time and money to the application of proven procedures to decrease infections and other complications of hospital care.
You cannot know best practice based on individual experience. I like to tell the residents that the three most dangerous words in medicine are “In my experience.” You need large numbers of patients and studies to guide practice. The last 15 years have seen a large number of clinical trials aimed at discovering what is the best practice to prevent everything from line related infections to deep venous thrombosis. Dozens of science based investigations whose goal was to improve patient care in the hospital, and my hospitals aggressively applied them.
The first intervention was the use of alcohol foam instead of hand washing. I have in my mind, and cannot find the reference, that if a nurse would wash her hands appropriately after every contact, he would spend 80% of their shift washing hands. Soap and water, it turned out, was not a practical solution to keeping hands clean. It is too time consuming in a busy work day, despite it’s proven efficacy in preventing infections.
Alcohol foam can be used in a fraction of the time with superior results since it is much easier to foam frequently. And once I discovered it was not to be used orally like cheese whiz, the results were even better.
The foam is now ubiquitous in the hospitals. Even when the use of the foam was 20%, the overall infection rate in the hospitals fell by half when compared to rates with hand washing. Then, over the next decade, the hand hygiene compliance rate has steady increased to around 90% and there was a corresponding steady decrease in infections. It took over a decade of consistent work and a lot of trial and error to get the rates to 90%.
At one hospital the limiting problem was no foam outside the rooms. No one would walk an extra few feet to get to the foam. But at another hospital the fire Marshall said alcohol foam in the halls was a fire hazard and we could not put alcohol in the halls. He was eventually overruled, but what are you going to do in the meantime?
And I could go on for paragraphs about the issue of finding product that minimized the number of HCW’s whose hands where turned raw by the alcohol.
Ninety percent seems to be the best we can consistently achieve with the current program for hand hygiene, and we are puzzling over how to get the rates to 100%. One approach is the “It’s ok to ask” program, where patients are encouraged to ask their provider if they washed their hands. I asked a series of patients if they would ask, and they uniformly said no, they did not want to risk angering their health care provider. I agree. It is important not to piss off the person providing your morphine. Besides, would you fly on an airline if their motto was “It’s ok to ask if the landing gear is down.”
Last year I ran a red light. It was 7 am, I was taking the kids to school and I have to make a right then an immediate left across four lanes of traffic. I make this turn everyday. I am talking to the kids and I look several times, no traffic, and make the turns.
What I did not see was that the light was red nor did I see the cop stopped on my left. I was so intent on the traffic I missed two key features in my environment.
It is the main reason, I suspect, that we cannot get hand hygiene to 100% every time, every where. The hospital has too many opportunities to focus our attention elsewhere that, for the short term, allows us to forget to foam.
Somehow, and I do not know how, I suspect we need to make foaming the default rather than optional; then our rates will get to 100%.
But foam is not the only intervention my hospitals have implemented.
Surgical check lists, best practice bundles (collections of proven interventions gathered together) to prevent ventilatory pneumonias, to prevent intravenous catheter related infections, to prevent urinary tract infections, to prevent deep venous thrombosis. Innumerable checks and balances with pharmacy to prevent medication errors.
Simple things to prevent surgical wound infections but logistically difficult to get to 100%: timing of antibiotics to within an hour of cut time, no shaving the surgical site, not letting the patent get cold post op, and tight glucose control were are associated with decreased wound infections. Next up may be no staples with orthopedic cases as a recent meta-analysis demonstrated fewer infections with sutures. That will be fun, getting surgeons to alter practice.
Over the last year my hospitals have implemented dozens of practice improvements based on the medical literature to improve outcomes and the results have been amazing. Practices that were not effective were abandoned or modified, sometimes going through multiple iterations until were discovered was worked and was practical.
As a result, at Legacy we have prevented over 200 deaths (12.5% reduction in non-risk-adjusted mortality rate, which is now 1.47% for our system that includes a regional trauma center and regional burn center as well as two NICUs, oncology program, and multiple other high-risk programs) and over 570 prevented infections (39.5% reduction in whole-house infection count) above historical data. And that is over the most recent 24 months. At Legacy it is estimated we have also saved 8 million dollars in associated costs.
A few of the hospitals have gone a year without a ventilator associated pneumonia or a catheter related infection. Every year has seen a decrease in the healthcare associated infections and other complications.
That is 100 deaths prevented a year for 5 hospitals. Multiply that for the remaining 4,995 hospitals in the US and the IOM estimates for last century seem reasonable. But not for this century and not for the decades to come.
I used to think that infections were inevitable, but no longer. There is the occasional patient who will get an infection: the badly burned, the multiple trauma. But even the trauma ICU had a marked decrease in all infections with increased infection control compliance. We had a wound infection in a 400 lib patient who literally had dirt tattooed in the palms and soles and a Hemoglobin A1c of 15 who required emergency surgery. I was not surprised that patient developed an post-op infection. We did everything correctly and still had a complication. Sometimes the barriers we have to overcome to prevent infection may be too great, but it does not stop us from trying.
But the experience of Legacy and Adventist demonstrates that aggressive adherence to proven infection control works and that the majority of health care associated infections and deaths need not happen.
I have three epiphanies in my life: my first great meal (at a restaurant called St. Estephe’s), my first great Bordeaux (oddly enough, a St. Estephe’s), and when I realized that most infections in the hospital need not happen.
This has been a real decrease in infections and death, not just playing with numbers to look better. These hospitals look at every healthcare associated infection (HAI) as an improvement opportunity and do not sweep data under the rug.
I also know personally that the numbers are real. I used to derive a significant portion of my income from hospital acquired infections. There are many reasons why my income has declined by 60% over the last decade, not the least of which being large numbers of patients that used to make up my practice (HAI’s, AIDS) have disappeared. The majority of those 570 prevented infections would have been consults. I feel like Phillip Morris making stop smoking ads.
It was not easy making these changes; it took years of committed work. People are like oil tankers and change course slowly. And some are filled with toxic waste. An interesting aspect of instituting the policies and procedures has been who fought against the changes the hardest. Docs. Not all of them, just a small subset. There is a curious subset of MD’s who feel that the data does not apply to them. They do not need to follow infection control procedures, use full barriers when placing a line, or even wash their hands. And I do not get it. I cannot figure out why some docs are so recalcitrant about doing the right thing, the proven thing. Eventually everyone complied, but some people made it more painful than it needed to be.
My hospitals made a serious commitment to providing the best care as determined by the science. It was not simple and required a surprising amount of creativity and time to apply the evidence to the real world. But the nice thing is that when you apply science to problems, you get results. Science works. Quality initiatives work. Next time you point out the deaths caused by modern medicine, leave my hospitals out of it.
It makes me wonder. There are numerous naturopathic, chiropractic, and other alternative schools and clinics involved in patient care. I am sure that they too have numerous quality improvement studies to brag about that have improved patient care and outcomes.
Think of all the practices in medicine that, eventually, have been demonstrated to be worthless, or dangerous, or flawed and that were improved or abandoned for the betterment of patient care.
So let’s start a list, shall we. The following is the top 10 list of alternative medical practices that have been modified or abandoned because of studies that demonstrated they were ineffective or dangerous and the quality initiatives that have improved patient care:
1) Disposable acupuncture needles (thanks to wales)
2)
3)
4)
5)
6)
7)
8)
9)
10)
Sorry. I found nothin’.
Perhaps it is a reflection of the perfection that is alternative medicine. Alternative medicine practices change based on evidence? No need.
Of course, I may well be wrong. As the board president of the Oregon Association of Naturopathic Physicians states, “Both MDs and NDs are trained to work from the evidence-based model of medicine, using best practices and standards of care.” I suppose my inability to find examples is due my inadequate Google and Pubmed skills to find the readily available information. I would have my 13 year old do it for me, but he is on a trip. Even if only 25% of medicine is science based, that is still 25% more than alternative practices.
Please, please, please, someone show me up. Hell, just give me some hand hygiene improvement data and let me know that, if nothing else, there is an understanding of germ theory in the alt med world. Ever since my local paper, the Oregonian, printed a picture of the local Natural Medicine School teaching acupuncture without gloves, I am not so sanguine about that understanding. It still gives me the willies to see that photograph and it looks for all the world that there are two boxes of gloves in the background, so I know they have them. It may be that all the gloves are left handed or right handed and so cannot be worn. Sometimes I pull out a glove for the right hand and it is a lefty glove and then I pull out a glove for the left hand and it is a righty, so I cannot find a pair to wear. It’s a problem.
Medicine slowly improves, too slowly sometimes. I know that 20 years ago we did not have the information to inform our practice that we do now. We did the best we could with what we knew at the time, and we do the best we can now with the information we have today. Still, despite the impressive improvements, it is a bittersweet victory. I can’t help but think what could have been, if only we had known.

Is something better than nothing? The many ineffective ways we treat nocturnal leg cramps
While leg cramps won’t kill you, they can make you miserable when you are trying to sleep. There’s not much evidence for effective treatments, and there are far more proposed treatments than there is evidence.
Scott Gavura on August 10, 2017
Link

If you’ve ever experienced the pain of a charley horse, then you’ll have an idea of what about half of all adults over the age of 60 experience – nocturnal leg cramps. A nocturnal leg cramp can occur in the leg or foot, and is a painful and sudden tightening of a muscle that can last from seconds to minutes, and can recur in the night. Not surprisingly, nocturnal leg cramps can interrupt and impair sleep and while these cramps don’t cause permanent injury, they can lead to decreased quality of life from the pain and sleep impairment. Despite the frequency with which these leg cramps occur, very little is known about what causes them and what anyone can do to prevent them. Consequently, there are an enormous number of medicinal, folk, and alternative remedies that are used and endorsed by both health professionals and patients alike. And a recent survey has shown that most people are treating leg cramps with remedies that are useless, dangerous, or useless and dangerous. What’s the best approach when there’s no decent evidence?
The causes of nocturnal leg cramps are not well understood. The cramp itself is due to the repetitive firing of motor units in the muscle but what triggers the contraction is not known. While often attributed to electrolyte imbalances or dehydration, there seems to be no relationship between the two. Many drugs are associated with reports of leg cramps (even drugs used to prevent leg cramps), but there’s very little evidence to show that any actually cause them.
Unless an underlying medical cause has been identified for nocturnal leg cramps, the approach to idiopathic cases focuses on reducing their frequency and severity. The acute management of a leg cramp is the most immediate concern, and usually stretching the muscle is enough to stop the cramp. Stretching to prevent leg cramps is the most commonly recommended non-drug approach, and while stretching has been studied in clinical trials, the evidence is mixed, with controlled trials (sham stretching) reducing the observed effectiveness of this approach.
Perhaps not surprisingly there are an enormous array of supplements and remedies used to prevent leg cramps. A recent survey of treatments used by 632 older adults with leg cramps in medical practices in Alsace, France illustrates the variety of approaches in use. This was a questionnaire-based study that evaluated if patients had nocturnal leg cramps, if they used any therapies, and what those therapies were. Of the 632 patients, 133 (20%) were using drug- or non-drug treatments to treat about 3 episodes of cramps per month. I’m not going to review the statistical analysis in any detail as I think the breadth of what is studied is what is interesting, rather than their absolute ranking. Here’s the list of treatments used:
It’s quite the list. Magnesium is number one, despite evidence that it is no better than placebo.
Quinine is number two, which perhaps reflects cultural differences, as quinine use for leg cramps has dropped elsewhere as a result of FDA action against its use. Quinine seems to have some effectiveness against leg cramps, although the evidence is of mixed quality. But what is well known is that quinine can cause fatal allergic reactions as well as a host of other toxicities, even after a single dose. After the FDA issued a warning about quinine and implemented new prescribing requirements in 2010, usage dropped 99% in the USA.
Again, perhaps representing cultural and geographic trends, homeopathy was reported as number three. Not much can be said about the elaborate placebo that is homeopathy, but perhaps the best that can be said is that it doesn’t have the potential to kill you, unlike quinine. (Amazingly, a homeopathic leg cramp “remedy” has been endorsed as the official cramp relief sponsor of the Boston Marathon.) The rest of the treatments identified by patients in this sample range from anti-inflammatory drugs to antidepressants and even to extract of leeches, a French remedy that has yet to receive FDA approval.
Among the non-drug treatments, what appears to be a folk remedy (soap in the bed) is #1, followed by an array of approaches that seems to be endorsed or accepted by most of the treatment guidelines that exist: stretching, mobilization, heat and massage all tend to be recommended, usually with the caveat that there is little good evidence to show that they are effective.
Choosing wisely when there’s no good evidence
While nocturnal leg cramps affect millions, there’s little good information available to guide patients or health professionals in choosing treatments. The wide number of approaches that are used reflects the reality that nothing really works effectively, so any benefits perceived are more likely to be random variations in their occurrence rather than actual cause and effect. The absence of evidence is frustrating, given how common and debilitating nocturnal leg cramps can be. This doesn’t appear to be a big area of research, so we are likely to be in this situation for the foreseeable future.
When faced with a lack of good evidence for any treatment, we should then focus our attention on safety. Non-drug measures like massage and stretching would seem to have the lowest risk, even lower than supplements like magnesium or vitamins. Given the poor quality control pervasive in the manufacturing of herbal remedies, and given the lack of evidence they work, they would appear to also present an unfavorable risk-benefit ratio. Only when non-drug treatments have been tried, and only when leg cramps are truly disruptive to quality of life, would it seem that drugs (or supplements) would be worth test. Until better evidence emerges, it seems many of us will continue to struggle with preventing and treating nocturnal leg cramps.

The appeal of being a medical “apostate”
There has long been a huge appeal in medicine that derives from being an “apostate”. Since COVID-19 hit, apostasy has become like a drug among too many doctors, and social media has amplified the popularity of “medical apostates” beyond anything I’ve seen previously.
David Gorski on November 7, 2022
Link

Recently, I was taking my turn supervising the surgical skills lab for the interns in the residency program in which I’m faculty. It was a practice session for knot tying, suturing, and using the simulators to practice basic laparoscopic skills, as the course was nearing its end and the final examinations are coming up. I happened to be talking with some of the interns as they practiced or took breaks, and somehow the topic of medical misinformation and antivaccine physicians came up. As I’ve noted a number of times among my colleagues and trainees, one intern was surprised at how many physicians have been promoting antivax disinformation and COVID-19 quackery since the pandemic hit. As I am wont to do, I pointed out that we’ve always had these “contrarian” doctors in our profession’s midst; it’s just that most of our colleagues didn’t notice or pay attention to them. Many didn’t believe they existed—or thought our complaints about them were exaggerated—and as a result there has been little impetus in our profession to do anything about them, a task that specialty boards and state medical boards have historically been depressingly lax about. Indeed, it’s a topic that I’ve written about almost as long as I’ve been blogging, and way back in 2008 Dr. Val Jones even coined a term for doctors who didn’t pay much attention to medical pseudoscience, the “shruggie“. I noted that no one who paid attention to these issues before the pandemic has been particularly surprised to see how many quacks and antivaxxers are in our midst. Indeed, some of these interactions are why of late I’ve been stepping back a bit and writing more and more about my views regarding why and how physicians ostensibly trained in science-based medicine break bad and use their credentials to attack the science-based regulation of medicine, viewing it as “thoughtcrime” and “censorship”.
In 2009, I laid down some of my thoughts on why doctors become quacks and updated those thoughts a couple of weeks ago, but I now realize that even that update was incomplete. Although I did discuss hubris, arrogance, and ego gratifications as some of the traits that fuel the conversion of a physician into a quack and antivaxxer, an article in The New York Times by Andy Kroll published a week ago led me to realize that I forgot one other important trait: Apostasy. So, even though there are a number of pieces of misinformation and disinformation that I could have written about, such as the resurrection of the SARS-CoV-2 “lab leak” conspiracy theory by Vanity Fair and ProPublica (the latter of which made me think that Kroll, who is a reporter for ProPublica, should turn his gaze on his fellow reporters), Robert F. Kennedy Jr.’s new conspiracy “documentary” based on his book on Anthony Fauci, or Emily Oster’s risible suggestion of “amnesty” for public health officials who “got it wrong” early in the pandemic that ignores the disinformation promoted by her allies and her, I kept coming back to apostasy for this week’s topic. Kroll’s article isn’t just about medicine, of course; it’s about an entire right-wing narrative about the pandemic. While it’s impossible for me to completely avoid mentioning politics, I’m going to try to deemphasize that aspect of this discussion, because the appeal of “apostasy” in medicine long predates the capture of the anti-public health movement including the antivaccine movement by the right. It’s a more general phenomenon than the politics of today might lead one to believe.
Kroll starts out with the example of a physician, one whom I have as yet not written about but one who is running as the Republican nominee to become governor of Minnesota, Scott Jensen:
On Apr. 8, 2020, in the chaotic early days of the Covid pandemic, the Fox News host Laura Ingraham welcomed a little-known state senator onto her prime time show. With his unmistakable Minnesota accent and an aw-shucks bearing, Scott Jensen, a Republican, was the furthest thing from the typical fire-breathing cable news guest. But the message that he wanted to share was nothing short of explosive.
He told Ms. Ingraham that he believed doctors and hospitals might be manipulating the data about Covid-19. He took aim at new guidelines issued by the Centers for Disease Control and Prevention, warning that they could lead medical institutions to inflate their fees. “The idea that we are going to allow people to massage and sort of game the numbers is a real issue because we are going to undermine the trust” of the public, he said.
Ms. Ingraham’s guest offered no evidence or data to back up this serious allegation. Coming from a random state senator, the claim might have been easily dismissed as partisan politics. What gave it the sheen of credibility was his other job: He is a medical doctor.
If there’s one thing that the pandemic has taught us, it’s that there is much hubris among my colleagues, in which physicians in specialties unrelated to public health, epidemiology, and pandemics come to think that they know better than experts in those fields. Dr. Jensen is no different. He is a family practice doctor who’s run a practice in Watertown, Minnesota, since 1986. He shows no evidence of possessing the knowledge, skills, and expertise relevant to making pronouncements about COVID-19, such as the ones that he made on Fox News and antivaccine outlets such as Del Bigtree’s vlog, such as the claim that people died “with COVID, not of COVID” and death certificates were being rigged to exaggerate the number of COVID-19 deaths. For instance, here he is in a Q&A with Minnesota Public Radio in March 2022:
Do you dispute that more than 12,000 Minnesotans have died as a result of COVID-19? Or that COVID-19 has killed more than 960,000 Americans?Yes, I do. I think that what we have, when you say killed, I would say that COVID-19 may well have played a part. I would say that if this is a person who’s dying of stage 4 colon cancer, and COVID was diagnosed in the last 48 to 72 hours and it was put down as a COVID death I think that’s problematic. And I think the only way to solve the problem rather than call one another names or accuse someone of misinformation or disinformation, why don’t we just do an audit?
When questioned about his expertise, he responded:
I studied epidemiology in 1976 and ’77 when I was in dental school. My wife was an immunology researcher, prior to veterinary school. My family is very medically oriented. I went back and did advanced studies as a part of a Bush Fellowship and epidemiology and leadership and communication were part of that. But I don’t think that’s really the issue. I think that you’ll find people a lot more credentials than me on both sides of this argument. And I don’t think that it necessarily does anything to constructively move forward to simply denigrate this group of people’s credentials and elevate someone else’s.
So he studied epidemiology 45 years ago as an undergraduate, and was awarded a Bush Fellowship in Leadership and Policy Studies in 1999. I note that it wasn’t difficult just to search the Bush Foundation website and find Dr. Jensen’s name. What did he study? According to the Bush Foundation, he studied “dermatology, allergy, computer applications, leadership, and ethics”. No doubt that’s why he moves on to a “both sides” fallacy:
I mean, you’ve got people like Jayanta Bhattacharya, Harvey Risch, Scott Atlas, John Ioannidis, Peter McCullough, these people are all highly regarded giants in their field. And then you’ve got other people like Tony Fauci and Mike Osterholm, and others that are respected on another side.
Of course, many of those names are quite familiar to regular readers of this blog, as we have written about them and how they have promoted COVID-19 contrarianism and minimization going back to very early in the pandemic (just use the Search Box for this blog if you’re interested in the details). I’m not going to dwell on that or relitigate past discussions. Rather, I want to move on to this part of Kroll’s article.
My first experience with medical apostasy: Dr. Robert S. Mendelsohn
I’ve frequently made allusions to the religious language that antivaccine doctors and doctors who have come to believe in alternative medicine quackery use. They love to dismiss conventional science-based medicine as “dogma”, as though it were simply another competing religious belief to theirs. Indeed, a number of physicians with views outside the mainstream who are not quacks and antivaxxers sometimes like to invoke the very same language, although rarely as aggressively as doctors like Dr. Jensen.
The apostate evokes images of a distinctly religious variety. The fourth-century Roman emperor Julian, who pushed to abandon Christianity and return to paganism. Freethinkers tortured and burned at the stake for daring to question the official orthodoxy of their era. And yet for as long as the word “apostate” has existed, it has possessed a certain allure.
To become one requires undertaking a journey of the mind, if not the soul, a wrenching transformation that eventually leads one to reject what was once believed to be true, certain, sacred. It’s a journey that often results in glorious righteousness. They’ve experienced an awakening that few others have, suffered for their awakening and now believe they see the world for what it is.
Although Kroll points to examples of politicians such as Ronald Reagan, who was famously a dedicated New Deal Democrat before he became a Republican, in particular:
“I didn’t leave the Democratic Party,” Reagan liked to say. “The Democratic Party left me.”
This was a clever bit of sloganeering by the future president. It was also the testimony of an apostate.
My first experience with medical “apostasy” dates back to medical school. My girlfriend at the time gave me a book by a Dr. Robert S. Mendelsohn, Confessions of a Medical Heretic, which had been first published in 1979. Note the language: Heresy and apostasy often exist in the same person, if that person is someone who was once a member of a religion who embraced beliefs considered heresy by that religion to the point of leaving and becoming an apostate.
Indeed, in the introduction to Confessions, Dr. Mendelsohn declared:
I do not believe in Modern Medicine. I am a medical heretic. My aim in this book is to persuade you to become a heretic, too. I haven’t always been a medical heretic. I once believed in Modern Medicine.
And there you have it. Dr. Mendelsohn was not just a “medical heretic.” He was a medical apostate, too, having renounced his “belief” in Modern Medicine for another belief system, while portraying patients in starkly religious terms:
Without the ritual of the checkup, internists would have trouble paying the office rent. How else can the doctor ensure a steady supply of sacrificial victims for the Church’s other sacraments without the examination? The Gospel said many were called and few were chosen, but the Church of Modern Medicine has gone that one better: All are called and most are chosen.
I remembered my shock at reading Dr. Mendelsohn’s jeremiad against modern medicine, particularly its religious language that cast Dr. Mendelsohn as the “heretic” who had discovered The Truth and medicine as a religion with doctors as its high priests. As these books frequently do, Confessions of a Medical Heretic mixed reasonable criticisms of how modern medicine operated with over-the-top rants that portrayed surgeons as bloodthirsty butchers who didn’t care if operations were therapeutic and necessary or not but did them anyway because they were greed-heads who just loved to cut. He was particularly harsh on obstetricians and gynecologists, whom he characterized as sacrificing women on the altar of surgery during childbirth. Not surprisingly, it turns out that Dr. Mendelsohn was antivax to the core as well, writing in a chapter in his 1984 book How To Raise a Healthy Child In Spite of Your Doctor that there was “no convincing scientific evidence that mass inoculations can be credited with eliminating any childhood disease”; “while the myriad short-term hazards of most immunizations are known (but rarely explained), no one knows the long term consequences of injecting foreign proteins into the body of your child”; and “there is growing suspicion that immunization against relatively harm-less childhood diseases may be responsible for the dramatic increase in auto-immune diseases since mass inoculations were introduced.” He even asked, “Have we traded mumps and measles for cancer and leukemia?”
If you read the article, you’ll find a number of antivax tropes that antivaxxers co-opt against COVID-19 vaccines today. Forty years ago, Dr. Mendelsohn portrayed measles and mumps as basically harmless diseases, with the vaccines being more dangerous. He pointed to outbreaks of pertussis among vaccinated children in much the same way that COVID-19 antivaxxers point to transmission of COVID-19 among the vaccinated, leading him to call mass vaccination against pertussis “indefensible”. (The pertussis vaccine is not a sterilizing vaccine and only partially inhibits transmission, even as it is good at preventing serious disease. Sound familiar?) He fear mongered about unknown “long term” effects of vaccines. He blamed the DPT vaccine for sudden infant death syndrome. He wrote this about the polio vaccine:
Meanwhile, there is an ongoing debate among the immunologists regarding the relative risks of killed virus vs. live virus vaccine. Supporters of the killed virus vaccine maintain that it is the presence of live virus organisms in the other product that is responsible for the polio cases that occasionally appear. Supporters of the live virus type argue that the killed virus vaccine offers inadequate protections and actually increases the susceptibility of those vaccinated.
This offers me a rare opportunity to be comfortably neutral. I believe that both factions are right and that use of either of the vaccines will increase, not diminish, the possibility that your child will contract the disease.
In short, it appears that the most effective way to protect your child from polio is to make sure that he doesn’t get the vaccine!
Everything old is new again. I know, I know, I write that too often, but it really is true.
Dr. Mendelsohn’s rhetoric was full of explicitly religious imagery, too. For instance, there’s a chapter in Confessions of a Medical Heretic called “Ritual Mutilations” (Chapter 3, about appendectomies, tonsillectomies, hysterectomy, cancer surgery, and more). While it was a reasonable argument regarding how many tonsillectomies and hysterectomies were truly necessary, that was not what Dr. Mendelsohn was about, as he referred to the “millions of mutilations which are ceremoniously carried out every year in operating rooms”. He also dismissed most cancer surgery as useless, predicting that most cancer surgery “will be regarded with the same kind of horror that we now regard the use of leeches in George Washington’s time”. Another example of religious imagery includes Chapter 7, “The Devil’s Priests“—in the “Church of Modern Medicine,” yet! If that’s not enough, he described hospitals as The Temples of Doom. Again, for every reasonable criticism about hospitals (e.g., the oversuse of laboratory and diagnostic tests of dubious value and fee-for-service reimbursement models that incentivize doing more procedures), there are several overblown ones that portray hospitals as abattoirs. (I exaggerate only slightly.)
Other chapters have titles like:
    • Miraculous Mayhem (Chapter 2)
    • Holy War on the Family (Chapter 5)
The religious language of “heresy” versus religion is overwhelming in the book, and it wasn’t just there either. In 1984, during an address to the “health freedom” group, the National Health Federation, Dr. Mendelsohn said something that really drilled home what I mean by how the view of medicine as a religion leads to quackery:
Doctors complain that quacks keep patients away from orthodox medicine. I cheer! Since all the treatments, both orthodox and alternative, for cancer, coronary heart disease, hypertension, stroke, and arthritis, are equally unproven, why would a sane person choose treatment that can kill the patient?
Both sides! Alternative or science-based medicine, they’re just two different belief systems that are “equally unproven”! Got it? Elsewhere, in the foreword to Immunisation: The Reality Behind The Myth by Walene James, he wrote:
Elegance of style is an additional bonus. For example, James compares modern vaccines—laden with formaldehyde, mercury, dog kidney tissue—with the “eye of newt and toe of frog” added to the brew of Macbeth’s witches. She wisely comments: “Is it too impudent to suggest that man has long had a love affair with decomposing animal proteins, noxious potions that ward off the demons of ill fortune?”
Immunization: The Reality Behind the Myth will open the eyes of those who still believe in the religion of modern medicine. It will strengthen those who have left that religion. And to protect every human being right from the start— this book is the most valuable gift you can present to the mother of a newborn baby.
Again, note the narrative of the apostate.
Also, note how Dr. Mendelsohn sounded rather New Age-y before Oprah Winfrey ever became popular, describing his alternative “new medicine” thusly:
Our New Medicine cuts across all political and ideological lines and touches the core of every person’s relationship with life: How long and how well will I live? The New Medicine, too, takes on some of the trappings of a religion.
And:
Faith is the first requirement for a religion, and you still need faith to practice the New Medicine. But you won’t need faith in technology or doctors, or drugs, or professionals.
You need faith in life.
By faithfully, religiously if you will, regarding life — and loving it — the New Medicine immediately will discredit Modern Medicine. The New Medicine need not come between a person and whatever traditional religion he or she chooses, because the religions that have survived all support life.
And:
Since life is the central mystery of our New Medicine, our “sacraments” acknowledge and celebrate the life of the universe. The “sins” of the New Medicine, in many cases, turn out to be the virtues of the Church of Modern Medicine: any practice that promotes or condones violence against life. The New Medicine says it’s a “sin” to restrict weight gain during pregnancy, to use the Pill freely on the theory that it’s safer than pregnancy, to submit to routine annual physicals, to put silver nitrate in babies’ eyes, to immunize children routinely, to be ignorant of nutrition, and a host of other activities that Modern Medicine promotes as “healthy.” These activities are sins not because they offend anybody’s idea of correct or polite behavior, but because they present a clear and present danger to life. They are offenses against biology. Since the life in our bodies seems to have an incredible capacity to heal itself, if given the proper conditions the corrective activities of the New Medicine — guilt and penance — will aim at producing those proper conditions. Imbalance is often as difficult to avoid in human life as balance is desirable. Since this is a human medicine, not one bound to the deathly formality of machines, hope is one thing that is never taken away from even the worst “sinner.”
The New Medicine doesn’t have any empty rituals. You fulfill the “commandments” and celebrate the sacraments by doing real things. Naturally, we have priests in this religion, too. But the New Doctor is not the prime mediator between the faithful and the object of faith. The authority of the doctor is severely limited by the individual taking the responsibility upon himself. Still, a system of ethics needs a mediator, a supporter of the faithful in their quest, a lifeguard when the quest runs into trouble.
I could go on and on, but you can read the entire book for free for yourself if you want, as a PDF of it is hosted at—where else?—Whale.to, which was a good thing for me writing this post, as I no longer possess the original copy of the book that I’d received in the mid-1980s. I really don’t know what happened to it.
I also can’t help but note that Dr. Mendelsohn was as underqualified to make such sweeping indictments of many areas of medicine as Dr. Jensen is to make sweeping indictments of public health responses to COVID-19.
Why is medical apostasy so attractive?
There’s no doubt that medical apostasy is very attractive to certain physicians. Nor is apostasy attractive only to the doctor who embraces “heresy” and becomes an “apostate” opposed to the “church of medicine,” as Dr. Mendelsohn so famously described himself, although maybe not using exactly those words. The idea of an apostate is attractive to those who would follow the apostate. In this concluding segment, I’ll look at both sides of this coin. After all, it doesn’t take much to think of a list of physicians who used the language of medical heresy/apostasy in describing their “conversion” to quackery; e.g., Andrew Wakefield, Paul Thomas, Stanislaw Burzynski, and many other physicians turned quacks have railed against the “dogma” of medicine that has left them on the outside.
In my last article on the making of COVID-19 contrarians specifically and quacks more generally, I noted that hubris was a key trait in physicians that lead them to believe that they know better than the assembled experts in their fields, that they alone have found some fatal flaw in the understanding of science missed by experts far more qualified than they are. In fairness, one has to recognize that it is always theoretically possible that a doctor such as Dr. Mendelsohn, Dr. Jensen, or any of the other panoply of “apostates” to see something that the experts didn’t. It’s just incredibly unlikely. Moreover, if such “apostates” brought home the evidence to prove their point, chances are good that eventually they could persuade the experts. I won’t claim that it would be easy or fast, but in general over time science does win out.
It takes a lot of self-confidence to believe that you have what it takes to be entrusted with people’s lives and most private information, as physicians are. This is even more true if you think you have what it takes to become a surgeon, the only profession to which society explicitly gives permission to remove or rearrange parts of people’s anatomy for therapeutic intent, to plunge into places in the body that even the person possessing that body can never see. Also, as I point out a lot, most physicians are not scientists. Many go into medicine and surgery because it’s a calling; they want to help people. Many go into medicine for religious reasons; indeed, unlike the case among scientists the level of religious belief among physicians is similar to that of the general public, at least in the US. It is therefore unsurprising (to me, at least) that many would view medicine and its practices in a religious light.
There is also a tendency among physicians to believe that they are the “best and the brightest”, because in large part many of us are, at least if you define “best and brightest” as academic achievement. After all, high academic achievement and strong work ethic are basically prerequisites to be accepted to medical school. As a result, when physicians find that the reality of medicine as a service, even an assembly line seeing patients if you will, there is often disappointment. For physicians who have always seen themselves as brilliant and creative (even if medicine itself tends to disincentivize too much creativity), as “brave mavericks,” as I like to call them, something “different” becomes very attractive. There’s a reason why there are medical pseudoacademic pseudoprofessional societies like the Association of American Physicians and Surgeons (AAPS), whose members seem to think they’re all even better than the children at Lake Wobegon, where all of the children were “above average“, as in unappreciated geniuses who are all “ahead of the herd”. Moreover, like the apostate, they are “persecuted,” as this old editorial titled “Medical Herdology” by a former AAPS president described:
Inescapably, the herd is a force to be reckoned with in all of our professional lives. We must be prepared to travel with it or alongside it, to one degree or another, without being trampled or singled out for extermination. And, for those few physicians who still believe in individual-based medicine practiced according to the principles of Hippocrates, and in watching out for one another when one of our own is attacked, fortunately we have the AAPS. We are a fellowship of “different doctors,” and the distinction is apparent.
It’s also very satisfying to be able to tell yourself that you were once deluded, but saw through the delusion and embraced The Truth, whether that narrative is accurate or not.
Going back to Kroll’s description of apostasy, these brave maverick physicians, in their own self-image, have “experienced an awakening that few others have, suffered for their awakening and now believe they see the world for what it is”. Of course, being an apostate can be good or bad. If you reject one harmful non-science-based belief system in favor of one that is science-based (as Jim Laidler did when he rejected antivax beliefs and “autism biomed” quackery), then being an apostate can be a good thing, and the good feelings about oneself the conversion engendered can encourage the apostate to continue to do good. However, when, as Dr. Mendelsohn and Dr. Jensen did, the “apostate” embraces a set of harmful beliefs after rejecting science-based medical conclusions, then being an apostate is not something to be applauded, even as that apostasy leads such doctors to view themselves as possessors of secret knowledge that their colleagues do not possess, refuse to understand, or even reject outright. Indeed, they become susceptible to conspiracy theory, because if everyone rejects what you believe to be knowledge that you possess that the vast majority of experts characterize as nonsense, then you have to explain why our knowledge is rejected. Enter conspiracy theory and characterization of a belief system as “dogma”. It never occurs to this type of apostate that their new beliefs are rejected because they are not supported by evidence; to them, it’s because of religious differences, with science-based medicine being cast as just another belief system, a sort of religion or dogma whose adherents are trying to cast “heresy” against their belief as “thought crime“.
Apostasy also provides a testimonial message that people tend to find very appealing. Kroll sums it up well in his NYT article:
There are few more powerful messages in human psychology than that of the apostate: “Believe me. I used to be one of them.”
I would add that there are also few narratives more powerful in reinforcing a change in belief system for the person who’s made that change. There’s a reason why an antivaxxer named J.B. Handley once likened Andrew Wakefield to “Nelson Mandela and Jesus Christ rolled up into one.” Particularly powerful to them was Wakefield’s narrative of “unjust persecution” for his views, just as the stories of martyrs are particularly powerful in religion. Moreover, apostates, because they have rejected beliefs that they used to hold dear, can be particularly strong in their new beliefs and persuasive in converting others. It’s very similar to how converts to a new religion are often the most devout.
As I like to say, it’s not the pure grifters who know they’re grifting who are the most dangerous. It’s the true believers whose belief leads them to grift.

   What does “anti-vaccine” really mean?
David Gorski
Link
We write a lot about vaccines here at Science-Based Medicine. Indeed, as I write this, I note that there are 155 posts under the Vaccines category, with this post to make it 156. This is third only to Science and Medicine (which is such a vague, generic category that I’ve been seriously tempted to get rid of it, anyway) and Science and the Media. There is no doubt that vaccines represent one of the most common topics that we cover here on SBM, and with good reason. That good reason is that, compared to virtually any other modality used in the world of SBM, vaccines are under the most persistent attack from a vocal group of people, who, either because they mistakenly believe that vaccines caused their children’s autism, because they don’t like being told what to do by The Man, because they think that “natural” is always better to the point of thinking that it’s better to get a vaccine-preventable disease in order to achieve immunity than to vaccinate against it, or because a combination of some or all of the above plus other reasons, are anti-vaccine.
“Anti-vaccine.” We regularly throw that word around here at SBM — and, most of the time, with good reason. Many skeptics and defenders of SBM also throw that word around, again with good reason most of the time. There really is a shocking amount of anti-vaccine sentiment out there. But what does “anti-vaccine” really mean? What is “anti-vaccine”? Who is “anti-vaccine”?
Given that this is my first post for SBM’s self-declared Vaccine Awareness Week, proposed to counter Barbara Loe Fisher’s National Vaccine Information Center’s and Joe Mercola’s proposal that November 1-6 be designated “Vaccine Awareness Week” for the purpose of posting all sorts of pseudoscience and misinformation about “vaccine injury” and how dangerous vaccines supposedly are, we decided to try to coopt the concept for the purpose of countering the pseudoscience promoted by the anti-vaccine movement. To kick things off, I thought it would be a good idea to pontificate a bit on the topic of how to identify an anti-vaxer. What makes an anti-vaxer different from people who are simply skeptical of vaccines or skeptical of specific vaccines (for instance, the HPV vaccine)? I don’t pretend to have the complete answer, which is why I hope we’ll have a vigorous discussion in the comments.
Believe it or not, I’m actually a relative newcomer to the task of taking on the anti-vaccine movement. Ten years ago, I was blissfully unaware that such a movement even existed; indeed, I doubt the concept would even have entered my brain that anyone would seriously question the safety and efficacy of vaccines, which are one of the safest and most efficacious preventative medical interventions humans have ever devised, arguably having saved more lives than any other medical intervention ever conceived. Even six years ago, although I had become aware of the existence of the anti-vaccine movement, I considered them a small bunch of cranks so far into the woo that they weren’t really worth bothering with. Yes, I was a shruggie.
All of that changed not long after I started my first blog in December 2004. Approximately six months later, to be precise. That was when someone as famous as Robert F. Kennedy, Jr. published an infamous screed simultaneously in Rolling Stone and Salon.com entitled Deadly Immunity that was so full of misinformation, pseudoscience, and conspiracy mongering that it altered the course of my blogging forever. Although I had already been becoming less and less of a “shruggie” about the anti-vaccine movement before RFK’s propaganda piece, “Deadly Immunity” resulted in a significant percentage of my blogging turning to discussions of the anti-vaccine movement and the scientifically-discredited myth that vaccines cause autism.
Anti-vaccine, not pro-safe vaccine
Before I try to define “anti-vaccine” in more detail, I should take a moment to point out that, if there’s one thing I’ve learned in nearly six years blogging about vaccines and the pseudoscience used to attack them, it’s that no one — well, almost no one — considers himself “anti-vaccine.” This is very easily verifiable in the outraged reaction elicited from people like J.B. Handley (who simultaneously gloats about the decline in confidence in vaccines among parents), Jenny McCarthy, and Dr. Jay Gordon when they are described as “anti-vaccine. Jenny McCarthy, for instance, will reliably retort, “I’m not ‘anti-vaccine.’ I’m pro-safe vaccine.” An alternative response is, “What I really am is ‘anti-toxins’ in the vaccines.” Meanwhile, Dr. Gordon will say the same thing while simultaneously saying that he doesn’t give a lot of vaccines and foolishly admitting in the comments of a blog post that some parents have actually had to persuade him to vaccinate “reluctantly.”
The rule that those holding anti-vaccine views will rarely admit that they are anti-vaccine is a good one, although there are exceptions. It is not uncommon to find in the comments of anti-vaccine propaganda blogs like Age of Autism and anti-vaccine mailing lists comments proclaiming explicit anti-vaccine views loud and proud, with declarations that “I am anti-vaccine.” This dichotomy has at times caused problems for the more P.R.-savvy members of the anti-vaccine movement, as demonstrated two years ago at Jenny McCarthy’s “Green Our Vaccines” rally, where images of vaccines as toxic waste and weapons of mass destruction were commonplace. Even so, the “Green Our Vaccines” slogan and coopting the “vaccine safety” mantle have been very effective for the anti-vaccine movement. In particular, Barbara Loe Fisher has successfully portrayed her National Vaccine Information Center (NVIC) as being a “vaccine safety watchdog” group looking out for parents’ rights, this despite hosting an online memorial for vaccine victims and a deceptive and disingenuous vaccine ingredient calculator.
“I know it when I see it”
In a concurring opinion in Jacobellis v. Ohio, regarding possible obscenity in a movie, Justice Potter Stewart once famously wrote:
I shall not today attempt further to define the kinds of material I understand to be embraced within that shorthand description ["hard-core pornography"]; and perhaps I could never succeed in intelligibly doing so. But I know it when I see it, and the motion picture involved in this case is not that.
From my perspective, defining what is “anti-vaccine” is a lot like defining obscenity: I know it when I see it. However, as in the case of pornography, even though it’s quite true that what is anti-vaccine is in general easily identifiable to those of us who pay attention to such matters, it’s much more difficult to define in a way that those who don’t pay attention to the issue can recognize. This difficulty is complicated by the fact that there are a number of different flavors of anti-vaccine views ranging from (I kid you not) the view that vaccines are a tool of Satan to depopulate the earth to much milder views. It’s also important to realize that most parents who buy into anti-vaccine views do so out of ignorance, because they have been misled, rather than due to stupidity. When I “go medieval” on anti-vaccine activists, my ire is almost always reserved for the leaders of the anti-vaccine movement, who spread misinformation.
When I’m actually in an exchange with someone whom I suspect of having anti-vaccine views, one rather reliable way of differentiating fear from real anti-vaccine views is to ask a simple question: Which vaccines do you think that, barring medical contraindications, children should receive? If the answer is “none,” then I’m pretty much done. I know I’m almost certainly dealing with an anti-vaccinationist. Be aware that this question may require some pushing to get an answer. Rarely am I able to get a definitive answer on the first try, because most anti-vaccine advocates are cleverer than that. They realize that I’m trying to get them to admit that they are anti-vaccine. Even so, if I ask something like, “If you had it to do all over again, would you vaccinate your child?” or “If you have another child, will you vaccinate that child?” I will usually get the candid response I’m looking for.
“Vaccines don’t work”? “Vaccines are dangerous”? They’re both!
If you look at the types of arguments used to oppose vaccination, they will almost always boil down to two different flavors, either that vaccines don’t work or that vaccines are somehow dangerous. Of course, we discuss the latter argument here all the time when we point out studies that refute the alleged link between vaccines and autism. Like the slogan “Tastes great, less filling,” both of these claims often co-exist to differing degrees, with some anti-vaccinationists arguing that both are true: Vaccines don’t work and they are dangerous.
This being the real world, one has to remember that vaccines are not perfect. They are not 100% effective, and there can be rare serious side effects. What differentiates anti-vaccine cranks from, for example, scientists who deal with issues of efficacy versus side effects and potential complications all the time, is exaggeration far beyond what the scientific data will support. For example, if the influenza vaccine is less efficacious than perhaps we would like (which is true), then it must be useless. This is, in essence, the Nirvana fallacy, wherein if something is not perfect it is claimed to be utterly worthless. Part and parcel of this approach involves the complement, namely vastly exaggerating the potential side effects and complications due to vaccines to paint them as being far more dangerous than the diseases they prevent. In addition, anti-vaccine activists frequently attribute harms to vaccines that the existing scientific data definitely don’t support as being reasonable or legitimate. The claim that vaccines cause autism is the most famous, but far from the only one of these sorts of claims. It’s not uncommon to hear fallacious claims that vaccines cause autoimmune diseases, asthma, and a general “weakening” of the immune system, among others.
One of the most famous examples of exaggerated harm or nonexistent risks is the infamous “toxin” gambit. This fallacious argument claims that there are all sorts of scary chemicals in vaccines. Of course, there are all sorts of chemicals with scary names in vaccines, just as there are all sorts of chemicals with scary names in almost everything, from food to clothing to household cleaners, among others. The dose makes the poison, and the amounts of these chemicals, such as formaldehyde, are tiny. As we’ve pointed out time and time again, for instance, the amount of formaldehyde in vaccines is so tiny that it’s overwhelmed by the amount of formaldehyde made as a byproduct of normal metabolism. Then, of course, there are the chemicals claimed to be in vaccines that are, in fact, not in vaccines. The most famous of these is undoubtedly the infamous “antifreeze in vaccines” gambit. Finally, there is the claim that there are “aborted fetal parts” in vaccines. This particular claim comes from the fact that, for some vaccines, the viruses used to make the vaccines are grown in a human cell line derived from an aborted fetus.
Overall, the “tastes great, less filling”-type anti-vaccine claims that vaccines are dangerous and don’t work, can be differentiated from scientifically valid concerns about the efficacy and safety of vaccines on the basis of how evidence is treated and the types of arguments that are used. Scientists, of course, tend to be a lot more measured and express the level of uncertainty in their claims; anti-vaccine activists are under no such constraints. When, for example, scientists debated how to respond to the H1N1 pandemic last year, there was considerable uncertainty about how to do it, when to do it, and how to formulate the vaccines. Which adjuvants? Should we use squalene in order to decrease the amount of antigen used? Contrast this to the anti-vaccine arguments, which tended to argue that H1N1 wasn’t harmful, that the vaccine was toxic and wouldn’t work, and even that it was a New World Order plot.
Approaching the evidence
If there’s one thing that distinguishes science from the way movements like the anti-vaccine movement approach evidence, it’s that the anti-vaccine movement values anecdotes over careful science. If you check out Generation Rescue’s website or any of a number of anti-vaccine websites, you will find numerous stories using the classic post hoc ergo propter hoc fallacy that in essence argues that, because one thing happened before another, that thing must have caused it. Because Generation Rescue preaches that its various forms of biomedical woo can “recover” autistic children, not surprisingly, there are a number of testimonials on its website touting stories of children who regressed after vaccination and then appeared to recover after whatever woo du jour was tried on them. On the surface, these testimonials seem convincing. However, given that millions of children who undergo vaccination each year, the number of vaccinations in the currently recommended schedule, and the fact that approximately 1% of children will eventually be diagnosed with an autistic spectrum disorder, it is not surprising, given the law of large numbers, that there will be a significant number of children who regress in fairly close temporal proximity to a vaccination by random chance alone. Even though such cases are random, though, to a single observer, they appear all the world as though the vaccine caused the regression. What’s difficult for non-scientists (and even many scientists) to accept is that it’s impossible to tell if vaccines are actually correlated with regression unless careful studies are done comparing large populations to determine whether children who are vaccinated really do have a higher chance of autism. Those studies have been done, and the answer is a resounding no. To the anti-vaccine movement, anecdotes trump evidence. Indeed, even physicians, like Jenny McCarthy’s son Evan’s pediatrician Dr. Jay Gordon, fall for touting their own anecdotal experience over careful epidemiology and science.
Perhaps one of the most characteristic aspects of the anti-vaccine movement is the same one that is shared by virtually every denialist movement, be it denying the science of climate change, evolution, or scientific medicine. That is the use of logical fallacies, cherry picking of the evidence, and distortion of the science. Perhaps the best example of cherry picking and distortion of evidence engaged in by anti-vaccine activists is a set of graphs I discovered earlier this year by a man named Raymond Obomsawin, in which he tried to represent as “proof vaccines didn’t save us.” As I pointed out in my response, this was intellectual dishonesty at its most naked. Basically, Obomsawin deceptively conflated mortality and incidence. Worse, he also chose his graphs in a way that parts of the data were left out. Indeed, less than a month ago, the Australian anti-vaccine activist Meryl Dorey approvingly cited Obomsawin’s graphs. When it was pointed out how Obomsawin had cherry picked his graphs to deceptive purpose, his response was beyond pathetic:
The software that I was using to create the graph did not allow for the creation of either a blank space or a dotted line between 1959 and 1968. There was no intent to be dishonest about this, and thanks to your blog, I will make it a point to specifically note on the graph that there is an absence of incidence data in this period.
I note that, not only have the graphs not been changed as far as I can tell, but Dr. Obomsawin is going to be giving a webinar tomorrow in which he is apparently going to argue the same nonsense. I’m half tempted to sign up and see what he says. It’d make for more blogging material, and an update to the Obomsawin Technique of vaccine denialism (yes, I named it after him) is probably overdue anyway. After all, I never took on several other of the graphs he included in his collection. My guess is that Obomsawin won’t change his graph. Obomsawin’s disingenuous approach to the evidence is mirrored by naturopath David Mihalovic, who wrote the infamous 9 Questions That Stump Every Pro-Vaccine Advocate and Their Claims. Dr. Crislip answered each and ever one of these “nine questions” without difficulty. The result was that Medical Voices (which published Mihalovic’s post) challenged us to a “public debate.” We said we’d be happy to debate them in blog posts but that a “public debate” would be counterproductive and allow them to engage in the Gish Gallup. The result? Like Brave Sir Robin, Medical Voices bravely turned its tail and fled. “9 Questions” remains on the impressively named International Medical Council on Vaccination’s website, formerly known as Medical Voices.
Another example of how anti-vaccine activists approach the evidence landed with a huge thud in the blogosphere last year when the Jenny McCarthy-fronted anti-vaccine group Generation Rescue decided to try to discredit the studies that show no link between vaccines and autism. As Steve Novella, Mark Crislip, and I all showed, Generation Rescue’s arguments were fallacious at best and deceptive at worst. The result was that J.B. Handley launched a broadside at Steve Novella.
Which brings us to another characteristic of the anti-vaccine movement.
Conspiracy mongering and reaction to criticism
Science is, if you’ll forgive the term, a highly Darwinian process. To be a scientist, you have to have a thick skin, because you’ll need it. Reviewers, mentors, other scientists, and virtually anyone to whom you present your results will be picking away at them, looking for flaws, looking for reasons to invalidate your conclusions. There’s nothing personal in it (usually); it’s how the process of science works. Similarly, those who have an appreciation for science understand that it’s a rough-and-tumble world where scientists have to be able to defend their work. Yes, it’s messy as hell, but it works. It may take a lot longer than we’d like and be a lot more confusing than the public likes, but over time hypotheses that don’t hold up are weeded out, to be replaced by those that do. Scientists understand this, and most know not to become too distressed by criticism. True, scientists are human too and can’t always separate themselves from their science; sometimes they lash out at criticism. However, for the most part, they don’t react the same way as denialists do when criticized. More importantly, contrary to scientists, it’s very rare indeed for an anti-vaccinationist to change his mind due to the evidence.
The anti-vaccine movement shares another characteristic with denialists of all stripes, and that’s an intolerance to criticism. Instead of answering it with science (which they can’t do), they tend to answer criticism with vitriol and conspiracy mongering. After all, when it’s all a conspiracy between the government and big pharma to “suppress” the data that allegedly show that vaccines cause autism, then anyone who speaks out for that viewpoint must be a pharma shill. If that critic is a woman, then she must have been the victim of a date-rape drug. Or he must be hopelessly compromised by relationships between big pharma and his university, even if such relationships need to be made up or extrapolated beyond all relationship to reality.
Perhaps the best way of describing how anti-vaccine groups react to criticism is to point out that their first response tends to be to try to suppress criticism rather than to answer it. Usually, this is accomplished through ad hominem attacks and poisoning of Google reputations. One particularly egregious example occurred when Age of Autism Photoshopped the faces of Steve Novella, Trine Tsouderos, Alison Singer, Paul Offit, Amy Wallace, and Tom Insel into a photo of people sitting down to a Thanksgiving feast of dead baby. Yes, likening enemies to cannibals wasn’t going too far. Sometimes attacks on critics can escalate to legal thuggery, which happened when the British Chiropractic Association sued Simon Singh and Barbara Loe Fisher sued Paul Offit. That such lawsuits nearly always fail, at least in the U.S., where the libel laws aren’t as draconian as in the U.K., doesn’t matter. The goal is not to recover damages; it’s to intimidate critics into silence. Unfortunately, even I’m not immune to being at the receiving end of this tactic.
Conclusion
Distinguishing true anti-vaccine rhetoric from cluelessness is not always easy. To help, I’ll recap the eight characteristics I’ve just discussed:
    • Claiming to be “pro-safe vaccine” while being unrelentingly critical about vaccines
    • The “vaccines don’t work” gambit
    • The “vaccines are dangerous” gambit
    • Preferring anecdotes over science and epidemiology
    • Cherry picking and misrepresenting the evidence
    • The copious use of logical fallacies in arguing
    • Conspiracy mongering
    • Trying to silence criticism, rather than responding to it
Someone who is anti-vaccine will almost certainly use at least three or four of these techniques. The cranks at Age of Autism use all eight and then some. Indeed, when these eight techniques fail to suffice, they make up more.
One other thing that’s important to mention, particularly since I’ve been guilty of this sin on occasion, is that we have to be careful about leaping to the conclusion that someone is anti-vaccine. That’s where the “I know it when I see it” test can backfire. For example, I was quite distressed at some of the statements coming from Dr. Marya Zilberberg on vaccines, in particular her referring to defenders of the current vaccine schedule as “rabid” and arguments that are reminiscent of the same sorts of arguments that the anti-vaccine movement uses for the HPV vaccine and the chickenpox vaccine, mixed with a question about combinations of vaccines being of concern that sounded uncomfortably like the “Too Many Too Soon” slogan beloved of Generation Rescue. Although I did not explicitly call her “anti-vaccine” in another forum and even went out of my way (as did Steve) to point out that I don’t think she’s anti-vaccine, I did try to point out to Dr. Zilberberg that, if you’re skeptical of some aspect of our current vaccination schedule, it’s important to be aware of how anti-vaccine cranks argue, so that you don’t inadvertently sound like one.
In the end, the anti-vaccine movement is another denialist movement, very similar to denialists of global climate change, science-based medicine, and evolution. As such, it uses many of the same fallacious strategies and distortions of science to promote its agenda and reacts the same way to criticism. Similarly, in the end, the anti-vaccine movement is also far more about ideology rather than science, which is why it remains so stubbornly resistant to reason and science. Finding an effective means to counter its message will likely require developing effective general strategies to counter science denialist movements of all types.
Let the discussion begin! What are the characteristics of anti-vaccine arguments and organizations that allow us to identify and characterize them as “anti-vaccine”?

   On the nature of “alternative” medicine cancer cure testimonials
David Gorski 
Link
No doubt you’ve come across them before, either on the Internet, printed advertisements, or radio and TV ads: Alternative medicine cancer “testimonials.” They are the primary means by which “alternative” therapies for cancer (or just about any other disease) are promoted and the primary “evidence” that is used to “prove” the efficacy of non-evidence-based therapies. There’s no doubt that they sure can sound convincing. Typically, what you will see or hear is a chipper-looking and -sounding person who claims that this treatment “cured” his or her cancer. These testimonials almost always include many or all of these elements: First, the cancer patient receives the diagnosis, after which she is lost and suffering at the hands of “conventional” doctors, who either cannot or do not wish to understand and who cannot do anything for her. Often, this will take the form of the classic alt-med cliche that the patient was “sent home to die.” Then, when all hope seems lost, the patient discovers an alternative medicine “healer” or treatment. It is not infrequently described in quasireligious terms, like a revelation or something that brings the patient out of the darkness and into the light. Naturally, there is resistance from the patient’s doctors, family, and/or friends, who warn against it, with doctors warning of dire consequences if the patient abandons conventional medicine. But the patient, convinced by dubious practitioners, friends, and, of course, previous testimonials, “sees” that the treatment “works” in a way that medical science cannot and survives. Infused with fervor, the patient now wants to spread the word. Often, the patient is now selling the remedy. Perhaps you’ve seen such testimonials or heard them on the radio and thought: “Gee, this sounds great. I wonder if it works.”
The answer is: Almost certainly not.
Let’s face it. Cancer is definitely scary. In fact, a cancer diagnosis is very, very scary thing for anyone to face, even when it is a cancer that is treatable and potentially curable. One reason is that it’s such a common type of disease that, by the time we reach a certain age, almost all of us have seen at least one friend or loved one die of it. All too often, that death is horrific, and even when it is not the wasting and weakness that are often seen before the end provokes a visceral reaction produced by few diseases. Worse, the treatments of cancer can be toxic. For solid tumors, surgery, sometimes disfiguring, is often required. Radiation therapy may be needed, or, most feared of all, chemotherapy. Contrary to what is said on many websites dedicated to “alternative” medicine, doctors do not “cut,” “burn,” and “poison” cancer patients (as proponents of unscientific treatments for cancer often characterize cancer treatments) because we enjoy it or because of failure of imagination. We use them because they are the best treatments that we currently have, and, for several common malignancies, they are quite effective. Even so, this cartoon remains a common misconception of what cancer therapies are.
Given the fear of cancer and treatments for cancer, it is not surprising that nonscientific “cures” flourish and many people, unfortunately, fall prey to them. Of course, it is not science that drives these sorts of therapies, although they are often dressed up in “science-y” sounding terminology in order to obscure what to an oncologist or cancer investigator would obviously be the extreme biological or even physical implausibility of the remedy. These treatments do not rely on randomized clinical trials or basic science to support their efficacy, either. Rather, they rely on testimonials, one of the most visible and highly abused methods of selling dubious therapies. Because it is my specialty, I will concentrate on breast cancer testimonials as a prototypical example, although many of the same principles apply to any “alternative medicine” testimonial.
Before I can explain why testimonials are generally unconvincing to anyone who knows a bit about cancer, a little terminology is in order. The treatment of breast cancer is divided into two phases, locoregional control (treatment of the disease in the breast and the axillary lymph nodes) and systemic control (prevention of distant metastases). Surgery and radiation therapy are modalities used for local control; chemotherapy and hormonal therapy, for systemic control. Adjuvant therapy is one of these modalities administered after surgery. Adjuvant radiation therapy will improve local control and lower the rate of recurrence in the breast. Adjuvant chemotherapy and hormonal therapy will improve systemic control and decrease the rate of development of metastases, which are usually what kill patients. Such adjuvant therapies are what I like to call “icing on the cake” of surgery.
The reason breast cancer testimonials sound so convincing is that most lay people don’t know a lot about the disease, particularly that surgery alone “cures” many breast cancers. Early stage cancers are cured by surgery alone more often than not, and a significant minority of patients with even large tumors and multiple positive lymph nodes still have a not insignificant chance of long term survival with surgery alone. I’ll start with a very common example. In the case of a lumpectomy, the local recurrence rate in the breast can be as high 30-40%. Radiation can reduce it to less than 10%. That means that women who forgo radiation are still more likely than not to avoid local recurrence in their breast, particularly if their tumor is small. In terms of distant metastases, which is what kills in the case of breast cancer and virtually all solid malignancies, chemotherapy and hormonal therapy improve survival. The improvement is small on an absolute basis in patients with early stage cancers but becomes more impressive with more advanced operable tumors. Because many breast cancer patients will do well with surgery alone, clinical trials with large numbers of patients are needed to find true treatment effects due to adjuvant therapies. Indeed, one of the hottest areas of cancer research is to determine molecular characteristics of tumors that will allow oncologists to differentiate women with low risk disease, who could forego chemotherapy safely, from those with high risk disease, who would be far more likely to benefit. As it is now, we give chemotherapy to the many in order to benefit the few simply because we do not know which specific women will benefit.
These facts help to explain breast cancer survivors who have undergone surgery but decided to forego chemotherapy and/or radiation therapy in favor of “alternative” medicine (Suzanne Somers, for instance). When such patients have a tumor with a good prognosis, where recurrence is uncommon or have a more advanced tumor but are lucky enough not to recur, they attribute their survival not to the primary surgery, but rather to whatever alternative therapy they have decided to take, even though it almost certainly had nothing to do with their survival. To them, it was the alternative medicine that “saved” them, not good old-fashioned surgery. In contrast, women who opt for alternative therapy and then recur obviously don’t provide good testimonials to sell alternative medicine, which is why you almost never hear about them. They’re bad for business.
Some might ask: Why do patients fall for this? It is not a matter of intelligence. In my experience, women who pursue alternative therapy are, more often than not, intelligent and/or highly educated. Instead, they do not possess sufficient scientific knowledge and/or critical thinking skills to separate truth from nonsense in medicine. Cancer and most other diseases are not something about whose therapy one can self-teach with much hope of success. However, because we all have to deal with health issues at one time or another, we all have opinions about them and, in a democratic delusion, tend to think that we can teach ourselves enough about the science of a disease to become “experts.” We don’t do this so much about, say quantum physics. Few would think that they could self-educate about particle physics enough to argue against well-supported science when it comes to particle physics, but many will do so with medicine. Unfortunately, we as physicians all too often encourage such thinking by being dismissive or not listening closely enough to the complaints of our patients.
It also seems to be a question of human nature. The diagnosis of breast cancer can be emotionally devastating. Formerly self-assured women feel themselves losing control of their lives. Unfortunately, our system of medicine reinforces this feeling of loss of control, as it is all too often impersonal and even disrespectful of patients. Patients find themselves going to multiple doctor’s visits, where all too often they have to wait for hours in crowded waiting rooms to see their doctors, who then, thanks to the demands of managed care, only spend 5 or 10 minutes with them discussing a life-threatening disease. They deal with voicemail hell trying to reach their doctor when they are having problems and endure many other indignities. They often conclude from this that the system does not respect their time or them (which it often doesn’t) and that they are considered nothing more than a number, a disease, an insurance number, or money. In contrast, “alternative” practitioners often provide the human touch that is too often missing from modern medicine. They take the time to listen to the patient and make her feel good about herself and her decision, all too often giving erroneous information about chemotherapy and radiation therapy. When a woman makes a decision to choose alternative therapy, she often sees herself as “taking control” of her treatment from uncaring doctors whose treatments, she is told, do not treat the root cause of her disease. Understandably, she may feel liberated from the tyranny of HMOs, cancer centers, clinics, and hospitals and back in control of her own destiny. In addition, many testimonials have religious overtones as well, where lost, suffering women misguided by conventional doctors and without hope find a savior (their “healer”) and/or enlightenment (the “alternative” therapy) that leads her out of the darkness and into the light of good health again. Her ignoring the reportedly dire warnings of doctors (unbelievers, apostates, infidels) is validated by her apparently excellent outcome.
That religion and spirituality should play such a large role in alternative medicine testimonials should not be surprising, given how much of alt-med is infused with New Age “spirituality” about living “energy flows” and connections with the earth. Consider, for instance, the concepts behind traditional Chinese medicine (TCM). These concepts are mostly based on a non-Christian religion (Taoism) particularly the emphasis of TCM on the need to correct “imbalances” between different kinds of spiritual “energies” in order to restore health. These concepts powerfully influence more of alt-med than just TCM. Sometimes, fundamentalist Christians, who would normally be very suspicious of such non-Christian concepts, manage find a way to infuse their brand of alt-med with their Christian religion (particularly faith-healing, which fits in well with alt-med spirituality) or to downplay inconvenient Eastern or pagan spirituality that underlies much of alt-med. (For examples of what I’m talking about check this and this out.)
Even doctors, who are trained to have the knowledge and critical thinking skills to know better, are not immune to falling under this spell. Case in point, Dr. Lorraine Day was Associate Professor of Orthopedic Surgery at UCSF and Chief of Orthopedic Surgery at San Francisco General Hospital in the 1980′s. She made a name for herself through dire warnings of AIDS spreading through aerosolized blood during trauma surgery (although, as far as I can tell from PubMed, she never published any studies in peer-reviewed journals to support her claims other than this interview). In the early 1990′s, Dr. Day developed breast cancer. Her website and this annotated transcript of one of her informercials tell her tale. In brief, in 1993 Dr. Day underwent an excisional biopsy that showed a ~2 cm breast cancer, with tumor extending to the margins. She then underwent what sounds like a re-excision lumpectomy, refusing the addition of axillary dissection, the standard of care at the time. (Axillary dissection has since been supplanted, for most breast cancer patients, by sentinel lymph node biopsy.) Dr. Day then started an alternative medicine regimen of diet manipulations and prayer. Nine months later, she developed a small “bump” near her previous site, which (she claims) grew to the size of a grapefruit in only three weeks. She even posted a picture. (I have to point out that I’ve never seen a breast cancer–recurrent or primary–even a really nasty one, that looked like this or that grew that fast. Invasive breast cancers usually start ulcerating through the skin long before they stick out and stretch the skin the way that Dr. Day’s lesion did.) The mass was, according to her, partially removed surgically, after which she was “sent home to die.”
Of course, this is not how her story ended. According to her testimonial, she “cured” herself with a regimen that included the same sorts of dietary manipulations and prayer as before. Dr. Barrett has posted a very good analysis of Dr. Day’s story and a deconstruction of her infomercial, concluding that the second operation most likely cured her and that the grapefruit-sized mass was most likely not recurrent cancer. Given that Dr. Day has refused to release the pathology report for her last operation after having released her first pathology report and only part of her second report (it’s a cancer staging form without the actual pathology report and leaves out the part that tells whether the residual cancer had been completely excised with clear margins at her second operation), I tend to agree with Dr. Barrett’s assessment. Very likely the last pathology report shows no breast cancer (in which case the second operation cured her) or a recurrent cancer that was completely excised (in which case the third operation cured her). Of course, Dr. Day could easily prove doubters wrong by releasing the the complete second report or the last pathology report, but she does not.
I mention this case to demonstrate that even highly trained and educated doctors, who should be able to evaluate alternative medicine therapies dispassionately, can become their biggest boosters. Even if Dr. Day could prove that she cured herself exactly as described on her website, I would still ask her why she never did a clinical trial to see if her result could be generalized to others, instead of using her story to sell a product called Barley Green and her books and videos. I would ask her why she didn’t hook up with basic scientists to study the constituents of Barley Green. That would be what a real academic surgeon would do. If her recovery was as miraculous as she claims, it would not take very many patients or very long at all to show the efficacy of her regimen. Unfortunately, Dr. Day appears to take a dim view of even honest criticism and is not above threatening her critics with the wrath of God. Ironically, recently Dr. Day announced that she will no longer sell Barley Green because the company that previously manufactured it was taken over by a Japanese company, which is going to supplant it with a barley product that uses “energized water.” Oddly enough, Dr. Day can believe that Barley Green and prayer saved her life, but is very skeptical about anything that even has a whiff of New Age about
Another aspect of testimonials is denial. There is a need to believe that the chosen “alternative” therapy is working, even when the evidence shows that it is not. One example of such a testimonial is a woman by the name of Kim Tinkham. If you read her story, it contains many of the same elements that Dr. Day’s did. There is the diagnosis, the shock, and then the seemingly brave decision to go against what three doctors recommended for her stage III breast cancer, namely a lumpectomy and axillary lymph node dissection, followed by chemotherapy and radiation, a very reasonable recommendation with a decent chance of saving her life. What’s missing are two things. First, Tinkham didn’t undergo a surgical excision of her tumor. Instead, she fell under the influence of a practitioner named Robert O. Young, who believes that in essence all disease, including cancer, is due to excess acidity and treats it all by trying to “alkalinize the blood.” Second, she spared us the cliche of her doctors “sending her home to die.” Here is one part of her testimonial, that really stands out:
"She can still feel the tumor just underneath the surface of her skin, where it will probably remain for a while."
"But she knows it’s harmless. Now, it’s simply her badge of honor – the reminder that she, Kim Tinkham, defeated cancer without any surgery, invasive procedures, radiation or chemotherapy."
Note that Tinkam tells us that the tumor is still there! Note also that no mention is made of whether the tumor has grown or shrunk. What is most likely going on is that Tinkham is fortunate enough to have a relatively indolent, nonaggressive tumor. It is probably slowly growing; so that in the ten months or so since her diagnosis it may not be obvious that it’s grown unless she regularly undergoes ultrasounds, mammograms, or MRIs, which would give an objective measurement of its size. So, if the tumor is still there and is not obviously shrinking, how on earth can Young tell Tinkham that her tumor is gone? Here’s how:
"A recent blood test proved that the stage three breast cancer diagnosed in February is absent from her body. But she really didn’t need a blood test to tell her that. Just hours after she had her blood taken for the test, and days before she knew the results, she was confident that her new lifestyle had allowed her body to fight the cancer."
I want to emphasize right here that there is no blood test that can tell a woman that she is breast cancer free. None. Unlike PSA for prostate cancer and CEA for colorectal cancer, tumor markers for breast cancer are notoriously unreliable. Moreover, even if she did have a blood test that told her she is tumor free, if the tumor is still there, she should pay attention to the tumor, not to any dubious “blood test” that a dubious practitioner represents to her as having any validity whatsoever in determining the status of her tumor. One of the aspects of unscientific practitioners is that they often have unscientific and unvalidated blood tests or other tests that “monitor” the tumor. They then tell patients that their tumor is getting better on the basis of these tests alone, even as the evidence is that it is not. They tell the patient she is getting better (or even “cancer-free”) right up until she becomes moribund.
This testimonial also demonstrates the religious or spiritual element that is often behind so much of “alternative” medicine. I did a search on her name, and it became clear in accounts of how Oprah Winfrey had Tinkham on her show to discuss her decision to treat her breast cancer with “natural” methods that Tinkham believes in The Secret. The same sort of wishful thinking that led her to fall for nonsense such as the “Law of Attraction” has led her to fall for the science-free nonsense that Young is telling her about her cancer. The bottom line is that, to those with little knowledge of medicine, testimonials can sound convincing. However, behind them there is usually a Secret-like element of really, really wanting to believe. Moreover, even someone without much medical knowledge but with halfway decent critical thinking skills and some skepticism should be able to come up with the right question, namely: How on earth can Tinkham claim that she is cancer-free on the basis of a blood test alone when her tumor is still there? Such a person will see this testimonial for what it obviously is, and that person won’t necessarily have to be highly educated to do it.
Never forget that alternative medicine testimonials exist largely for one purpose: To sell a product. Most of them are advertisements, nothing more. They are no more “unbiased” than pharmaceutical advertisements for their latest, greatest drug. In fact, they are worse, because at least the pharmaceutical companies have to be able to back up their claims with science and disclose potential adverse reactions in their ads. No such requirements exist for most alternative medical treatments, mainly because most of them claim to be supplements rather than medicines. The other problem with testimonials is that they don’t rise even to the lowest level of medical evidence, the anecdotal report. Anecdotal reports in medicine require a careful documentation of symptoms, lab tests, diagnoses, exact courses of treatment, and a patient’s response to treatment. Testimonials almost never present these elements in sufficient detail to judge whether the treatment actually did anything. There’s just no way of telling truth from exaggeration or fiction. Boiled down to the essence, these testimonials usually appear convincing for one of these reasons:
- The patient never had cancer to begin with. It was either an erroneous diagnosis by a legitimate doctor or a diagnosis based on an unscientific and unvalidated test used by the alt-med practitioner. This is more common than one might think (one example is pancreatic cancer), and I may do a post on this issue in the future.
- The patient underwent partial conventional therapy before abandoning it for alt-med (for example, an excision but no radiation or chemotherapy, like Suzanne Somers or Lorraine Day).
- The patient still has cancer but, like Kim Tinkham, has been told by her practitioner that she is “cancer-free.” If the tumor is slow-growing, this may be convincing for many months or even a few years.
In conclusion, be very skeptical of alt-med testimonials. If you look at them closely, you will often find that the patient did have significant conventional treatment; that the story is vague (often omitting, for example, the stage or histology of cancer or even whether the cancer was biopsy-proven); that there is no objective data, just references to other testimonials; or that the data mentioned either comes from alt-med websites selling a product rather than peer-reviewed medical journals or is a non sequitur about studies in peer-reviewed sources. They also have an afterlife, and it is not uncommon for cancer testimonials from patients who have died of their cancer to continue to circulate on the Internet long after the patient’s death, much like the undead. Finally, remember that conventional medicine is not above misusing testimonials in advertisements. Treat them with the same degree of skepticism. Look for the scientific and clinical evidence, not stories of great cures, regardless of the type of testimonial. If there is one principle I hope to impart here, it is that the claims of conventional medicine and alternative medicine should be treated the same and that they should be held to the same standard of both basic scientific and clinical evidence. I do not differentiate between the two when considering evidence, nor should you.

   A Guide for Confronting Patients
Steven Novella
Link
I sometimes lecture on science-based medicine to my colleagues and one of the most common questions I get is how to deal with a patient who expresses belief in unscientific treatments. The dilemma for the physician is that professionalism requires that we do not confront patients regarding their personal beliefs. We are there to inform and advise, not preach. And yet proper medical care is often hampered by unscientific beliefs on the part of patients.
David wrote previously about a case he reviewed in which a woman with a very treatable early stage of breast cancer opted for “alternative” treatment rather than the standard treatment, which carries a > 93% good outcome. As a result her cancer progressed horribly – but she clung to belief in CAM despite its obvious failure in her case. This story highlighted the fact that giving patients proper medical advice sometime requires confronting their false beliefs.
Unscientific and bizarre medical practices are in vogue and are increasingly infiltrating the medical system through a combination of misguided political correctness, stealth, and apathy. This is exacerbating the dilemma for science-based practitioners who are caught between the imperative to do the right thing in accordance with evidence-based guidelines and the default respect for the patient and the desire to maintain a therapeutic relationship.
In my experience, however, these two goals do not have to be mutually exclusive. An uncompromising but non-judgmental approach works very well.
Cross-Purposes
One of the more challenging aspects of ethical clinical practice is to balance various ethical and practical concerns that are at cross-purposes. For example, the need to give patients informed consent includes the requirement to give them a realistic sense of their medical condition. If a patient has a terminal illness, they need to be told so. Gone are the paternalistic days when a doctor would lie to a patient to protect them.
It is even more important to inform a patient of the consequences of their medical decisions. If a patient refuses a recommended treatment and they are likely to die without the treatment, they need to be told so in no uncertain terms.
At the same time we don’t like to scare patients unnecessarily and cause undue stress. Therefore we won’t tell a patient that we think they may have cancer when it is still uncertain. It’s better to wait for the test results.
But what if a patient declines to take a test that will indicate whether or not they have cancer? Then it may be necessary to inform them that the purpose of the test is to look for things that are potentially serious and treatable. If they still refuse, you may need to break down and inform them that they may have cancer and the test is needed to rule it out.
In other words – part of the art of medicine is knowing how much information to give a patient in order to help them make the right decision. Or, at the very least, if they are about to make a risky decision it becomes necessary to inform them in stark terms why the decision is risky. The desire to protect the patient from unnecessary stress takes a back seat to giving them the information they need to make an informed decision.
Confronting Beliefs
Along the same lines, health care providers should not confront the personal beliefs of a patient, but what if those personal beliefs are motivating them to make very bad medical decisions? Then, in my opinion, it depends upon the nature of the beliefs. Religious beliefs should be completely out of bounds. It is not the place of the physician to validate or refute personal religious beliefs.
Non-religious ideological beliefs are the same. The point is that the physician cannot impose their value system onto their patient.
That doesn’t mean that nothing can be done within the confines of professional ethics. If a patient refuses a life-saving intervention because of their religious beliefs it is reasonable to make sure that they fully understand the consequences of their decision. It is also reasonable to suggest that they consult with family members. You may even offer to refer them to a support group or even counseling.
This can be done in a completely non-confrontational and non-judgmental way. You can respect their religious belief and their right to make their own decisions, but just advise them to take advantage of available support structures in making such a huge decision.
Confronting Claims
I think where most of the confusion comes in is confusing the approach to personal values or ideological beliefs with the proper approach to medical claims. While beliefs and values are out of bounds, any factual claims are fair game – regardless of their motivation.
First it is important to recognize that patients who are in a doctor’s office have already made the decision that the expertise and experience of that physician is worthwhile. By consulting a physician a patient is tacitly giving that physician permission, even the obligation, to provide their unvarnished medical opinion. In fact, holding back on giving an honest medical opinion is a throwback to the old paternalism – it in essence says that the patient is incapable of handling the honest opinions of their doctor and they need to be coddled to some extent. In reality, in my opinion, failure to give an honest opinion is misguided and harmful – and I suspect may be designed to protect the physician from an uncomfortable confrontation more than the patient.
It is far better to be unapologetic in stating that one is a science-based practitioner (this article is written for the science-based practitioner) and that one’s medical advice is based upon the best scientific evidence available. I find that when patients come to my office, that is exactly what they want. They are very happy to get my honest opinion, even if it directly contradicts what they believe. I don’t dance around the issue at all.
Most experienced clinicians already know to avoid dismissive, insulting, or judgmental statements. In confronting unscientific medical beliefs, it is easy to do so. Simply confine your opinions to the scientific evidence. For example, if a patient asks me about acupuncture for migraine headaches I simply tell them that I have reviewed the published literature which does not support the use of acupuncture for migraines. I therefore do not recommend it. I am then happy to discuss the evidence with the patient as much as they desire. But almost always patients appreciate the fact that I have taken the time to actually read the literature and they respect my opinion. They may not follow it – but that is their choice.
Sometimes patients ask me about treatments that I believe to be fraudulent and exploitive – for example I am frequently asked about whether or not it is worth it to fly to China to get stem-cell therapy. In these cases I tell my patients, in a very factual and dry manner, that such clinics are fraudulent. What evidence we have shows that their treatments are not safe and that they do not work. In my opinion the people operating the clinic are committing fraud to steal money away from desperate patients. To do anything less is to fail to properly inform a patient.
I have even confronted the beliefs of patients that vaccines have caused their child’s autism – in some cases hardcore true believers. I validate the patient’s concerns and acknowledge the evidence that is out there. I reassure them that I simply want to give them the best advice I can, that I have read the literature thoroughly and then I give them my scientific opinion. It is often helpful to explain to patients how I approach evidence and my philosophy of science-based medicine. The response is almost universally positive.
Conclusion
Physicians and other clinicians should not be afraid to defend the scientific basis of good medicine and to explain to patients that we are scientific practitioners. In my experience there is still a great deal of basic respect for science in the public. As long as you take a thoughtful, professional, and evidence-based approach to your opinions, patients will respect them, even if they disagree with them.
It is profoundly misguided and harmful to fail to confront pseudoscience or bad science in medicine out of fear of offending a patient. Such fears are overblown and misplaced.
Given the medical environment today, clinicians are obligated to have a working knowledge not only of science-based medicine but of the unscientific practices and claims with which their patients may be faced. We are obligated, as part of good practice, to know the literature and the nature of unscientific claims and to help our patients navigate through them. Otherwise we are throwing them to the wolves.

Fooling Myself
Scott Gavura on June 2, 2016
Link


The first principle is that you must not fool yourself — and you are the easiest person to fool.
–Richard Feynman
I like to think of myself as a rational person, but I’ve been fooled by my own experience again and again. I’ve made bad decisions and wasted time and money believing what I was seeing, instead of being objective and looking at the evidence. One of my most memorable lessons has come over the past 14 years with my Labrador Retriever, Casey.
First the personal
We acquired Casey as a puppy, and she was less than a year old when she started limping. Investigations confirmed dysplasia, a genetic condition that leads to degenerative joints, arthritis, and pain. We were devastated. After considering the few treatment options that existed, we decided to skip surgery and treat it conservatively. I had no desire to start her on a lifetime of anti-inflammatory drugs, being very familiar with their side effect profile. I was familiar with a supplement used widely in humans that had some weak but somewhat promising evidence: We started giving her glucosamine and chondroitin supplements regularly. And we watched and waited.
It took some time, but Casey did appear to improve. We were thrilled. Life went on, and other than the occasional rough play session, Casey’s limping was mild, and she thrived. We continued the supplements, confident that we were doing good. But eventually I started paying attention to the emerging evidence on glucosamine and chondroitin. Once touted as a panacea for arthritis and joint pain, there had finally been some high-quality trials conducted – and the results were disappointing. Even this blog covered the issue, and contributors like Harriet were skeptical of glucosamine. Its supposed mechanism of action really wasn’t even that plausible. I started to wonder if the supplements were really doing anything for my dog’s pain. Eventually I decided on a trial – so I stopped the supplements about seven years after I started them. Neither my wife nor I could notice any difference at all in her mobility. Nor did the veterinarian. We’d been fooling ourselves, spending hundreds of dollars in the process.
Now I’m back in a situation with Casey where I’m making treatment decisions based on wishes and expectations, and not fact. The limping seemed to be managed with regular walks, and keeping her weight under control (some medication for a thyroid condition helped). But in the past few years, there have been changes. The rare use of meloxicam (an anti-inflammatory) gradually became a regular daily dose. The tail kept wagging, and we still had our long walks, but the energy started to decline. A trip to the veterinary emergency room a year ago led to a prescription for gabapentin (a drug typically used for neuropathic pain) which I’ve been reluctant to discontinue, despite the lack of good evidence it works, and even as her mobility has continued to deteriorate. Now she’s closing in on 15 years old. Casey’s legs are giving out on her, and she’s falling more and more. Just this week, she’s decided she doesn’t want to go for walks anymore, and she’s not finishing her meals – which never, ever occurs with Labradors unless something is seriously wrong. She’s becoming a shadow of the dog we’ve known. I think we’ve nearing a decision point – we’re about at the limits of what a dog’s body can do, and what veterinary medicine can offer. And while I know that all of this is to be expected in a Labrador Retriever who is 14 years old, I still want something, anything, that will give us more time.
And now the the professional
“The three most dangerous words in medicine: in my experience.”
-Mark Crislip
It’s exceptionally difficult to accept that you’re fooling yourself. As a pharmacist, I have spent a large part of my career speaking with people about health, illness, and self-care decisions. Some of most interesting discussions have been about homeopathy, because I know that any effects that patients may report are unrelated to the product itself – because homeopathy is effectively the same as doing nothing. Homeopathy is the air guitar of medicine – it’s packaged like a drug, and sold in pharmacies alongside actual medications, but the product are indistinguishable sugar pills, with no medicine in them at all. Rigorous trials demonstrate what common sense would suggest: Homeopathy is no more effective than an equivalent placebo. It’s not difficult to imagine the potential harm from someone using a product like a homeopathic spray for asthma. But beyond the avoidance of life-saving treatments, there’s the wasted resources and the overall illusion that fake medicine, like homeopathy, can create. A pharmacy I once worked in on a part-time basis had a sizeable homeopathy business back when homeopathy was more of a fringe product. I would never recommend the remedies, but was regularly asked for them. And I learned pretty quickly that it was difficult to convince regular users that they the beneficial effects they reported weren’t due to the remedies. Many were fiercely loyal to their particular product, for their particular ailment. They trusted their alternative medicine provider, usually a naturopath or homeopath. Suggestions that they might try reassessing them were rejected and sometimes even taken as insulting – after all, who was I to tell them (even gently) that the product had no medicine in it at all? From what they could see, homeopathy working for them, and that was all the evidence they needed.
As l started reading the Science-Based Medicine blog and began looking more critically at the evidence base, I changed my advice. With patients seeking arthritis and pain relief, I stopped sharing anecdotal evidence (“I give it to my dog!”) and focused on what the evidence was saying. I wasn’t all that successful in my dialogue with patients – or health professionals for that matter, who sometimes seem so reluctant to let go of what they already know to be true. If it’s not the belief that “the patient knows best”, it’s the accumulation of positive anecdotes that can turn health professionals into believers, even when the evidence says otherwise. I’ve spoken with pharmacists who see no contradiction discussing the molecular basis of drug interactions while simultaneously believe that water has some sort of magical memory. “I’ve seen it work” they shrug, “science doesn’t know everything.”
The reasons why we believe these anecdotes, despite the science, have been detailed at length in other posts. Add in the conflict of interest that that comes from making a profit, and it’s perhaps not surprising that many pharmacies sell products that lack good evidence of efficacy. And this is where I take issue with many of my peers who work in community (retail) pharmacy settings – I argue that selling implausible and ineffective treatments to consumers, given a pharmacist’s professional responsibility, is unethical. I object to the sale of dubious products in pharmacies because I believe that pharmacies are places of health care delivery, with a responsibility to the public that goes beyond basic commercial business ethics. (And as I’ve pointed out, pseudoscience fails that bar, too.) In an environment like a pharmacy, consumers may be more accepting. They may be in a vulnerable state. And once that belief in in place, it may be hard to shake it. Confirmation bias can be powerful, and it can lead health professionals and consumers alike into beliefs about health that are not substantiated when you take an objective look at all the data.
Who better to exploit the public’s trust in pharmacists than perhaps the pharmacy profession itself? In Australia, there’s a new trend to employ naturopaths in pharmacies to promote guide the selection of supplements and alternative medicine. With pharmacies held in high regard by the public, this move is expected to give more legitimacy to the alternative medicine system of naturopathy. And while naturopaths themselves may be well meaning, it’s been pointed out repeatedly at this blog that naturopaths take an extreme and very selective view of the scientific evidence. Naturopaths can give good advice, but this is when that advice happens to be congruent with the naturopathic philosophy. But naturopaths will routinely reject scientific evidence that doesn’t conform to their beliefs about medicine. If you ignore the non-confirmatory evidence about supplements and pick and choose what you want to accept as fact, then you’ll be more likely to believe that supplements offer good health benefits. Few do. This move has brought criticism from within and outside the pharmacy profession as a thinly-veiled attempt to boost sales:
The president of the Australian Medical Association Dr Brian Owler said pharmacists needed to re-evaluate their professional roles. “These things run the risk of diminishing the professional reputation of pharmacists and suggests they are more of a shop rather than a place to seek advice about medications,” he said.
Greg Turnbull, spokesman for the Pharmacy Guild of Australia, said if members employed complementary therapists, they should “be mindful of their duty of care and responsibility for any advice or service provided within their pharmacy”
Australian pharmacist Ian Carr is particularly critical:
“The problem is pharmacists need complete oversight of what goes on in the pharmacy,” he said. “You are in charge of training your staff appropriately, so that you know, right down the line the correct advice is being administered. “With a naturopath operating within your pharmacy they would need 100% supervision – but this is not the case.” Mr Carr said the uptake of naturopaths in pharmacies has been increasing due to public interest and also the industry not being supervised.
Carr is such a strong advocate for evidence in medicine that his pharmacy not only has no naturopaths, but he’s taken steps to help consumers make more informed choices. Like the United States and Canada, the Australian medicines regulator has two standards for health products – one which requires evidence, and one for products that lack evidence of efficacy. For the latter products, in Carr’s pharmacy they are sold with this sticker:

It’s not just the traditional alternative medicine – even some more conventional but very old products sold in pharmacies lack good evidence of efficacy, and are probably of questionable efficacy. Used fairly and consistently, this type of labeling supports consumer self-selection, but also emphasizes which products are backed by good evidence of efficacy. It’s also an opportunity for consumers to initiate a dialogue with pharmacists about the evidence, hopefully supporting good discussions about treatment options. In the long run, it may help consumers make health care decisions that are more informed by the scientific evidence.
Conclusion: The personal must give way to the empirical
Our personal experiences and anecdotes may be meaningful to us, but when it comes to health and medicine, they are a poor way to determine therapies and treatments that work. Recognizing that I was fooling myself with glucosamine and chondroitin was hard to accept, but it was an important lesson that has stuck with me and (I hope) kept me humble about the relevance of my own experiences and observations as a health professional. It’s also made me more sensitive to the perspectives of others in my dialogue with patients about their own perceptions of health. We believe what we see. We want to believe what we are doing is helping. And when we’re desperate, we want to do something…anything…rather than facing facts. And today, as I’m carrying my dog up and down flights of stairs, and watching her struggle to stand up, I’m struggling with that very issue.

Evaluating Treatment Claims: A Primer
Harriet Hall on July 15, 2014
Link
I recently wrote about the claim that acupuncture can improve vision in patients with macular degeneration. In response, I received this e-mail:
At Discovery Eye Foundation we have an education and outreach program for people with age-related macular degeneration, the Macular Degeneration Partnership. We are constantly getting calls from people who have heard of a new “cure” or a way to stop their vision loss. It is always hard to interject reality into the conversation and hear the hope leave their voice, replaced by frustration or despair.
The e-mail suggested I write an article providing guidelines for consumers to help them evaluate the validity of treatment claims for themselves. On SBM we are constantly stressing the need to apply critical thinking to what you read, and the many pitfalls to be avoided. I’ll try to synthesize some of the principles into a handy list of questions.
What kind of evidence is there?
    • If the claim is based on nothing but testimonials, STOP RIGHT THERE. You can forget about it, or at least file it away until there is better evidence. Anecdotes are not evidence; they only serve to suggest promising avenues of research. Science is the only reliable way to determine if a treatment is safe and effective.
    • Is the claim based on a gold standard randomized, placebo-controlled, peer-reviewed study or some lesser kind of evidence like case reports? Was it published in a reputable mainstream medical journal? (If you’re not sure how reputable the journal is, you can look it up on tables of “journal impact factors.”)
    • Keep in mind that half of all studies are wrong. There are many factors that can lead to error. Preliminary or pilot studies that are positive are frequently followed by better, larger studies that are negative. We can never rely on one study without confirmation.
    • Was it a meaningful clinical study in humans? In vitro lab studies and animal studies may not be applicable to humans; if promising, they must be confirmed in good human studies. If it was a human study, did it show meaningful outcomes that made a real difference, like a reduction in heart attacks, or did it just show an improvement in lab values or risk factors?
    • Were there 10 subjects or 300? Large studies are more trustworthy than small ones. The fewer dropouts, the better.
    • Did it use an appropriate placebo control that subjects really couldn’t distinguish from the active treatment?
    • Have other studies found similar results? Are there any studies that show the opposite? You can search PubMed and look for them.
Who is making the claim?
Is it someone who is likely to be biased? Is it someone who provides the treatment or sells the product? Is it someone with expert medical knowledge or someone like the schoolteacher who claimed she had invented a cure for the common cold? Does the person have a good reputation, or a track record of making questionable statements? Is it someone who quotes or associates with unreliable sources like Mercola.com, the Weston Price Foundation, or the Health Ranger? Quackwatch has a useful list of nonrecommended sources of health advice.
Where was the claim reported?
Real medical breakthroughs would be headline news. It’s not likely you would first hear about a cure for diabetes on an afternoon talk show or a Facebook page. Has your doctor heard of it? Are mainstream doctors recommending it? Is it covered on professional medical websites like the American Academy of Pediatrics or disease-focused websites like the American Diabetes Association? Is it sold only through multilevel marketing schemes?
What kind of language is being used?
Is it a sober factual report with caveats, or is it full of hype and buzzwords like “miracle,” “natural,” “known to the ancients,” “quantum,” “amazing,” “revolutionary”? If it really worked, advertising gimmicks wouldn’t be needed to sell it.
Does Gwyneth Paltrow swear it worked for her?
Testimonials can be very passionate but they are notoriously unreliable. Getting better when you use a treatment doesn’t necessarily mean you got better because of the treatment. Symptoms can fluctuate, diseases can resolve without treatment, and placebos can fool people. Perceptions can be wrong (think of optical illusions), the meaning of true perceptions can be misinterpreted, and memories can be inaccurate. Every snake oil salesman has reams of testimonials, and through the centuries there were testimonials galore for bloodletting to balance the humors. People frequently come to believe bogus remedies have worked for them. Barry Beyerstein wrote a classic article about that; it’s essential reading.
Does it make sense?
Is there a plausible mechanism of action? If someone claimed that standing on your head and whistling Dixie would cure diabetes, I think you would be skeptical. If it claims to work by a mechanism incompatible with known scientific principles, the level of evidence would have to be extraordinary for it to outweigh all the evidence those scientific principles are based on. Homeopathy’s claim that water can cure by remembering long-gone molecules, even after the water has been dripped onto a sugar pill and allowed to evaporate, would require extraordinary evidence indeed. If it’s a new antibiotic that is related to an old one, an ordinary level of evidence would suffice.
Is there a double standard?
Are they asking you to accept a “natural” or “alternative” treatment on the basis of the kind of evidence that you wouldn’t want the FDA to accept for allowing marketing of a prescription drug? There is only one science and only one standard of evidence.
Does it sound too good to be true?
Then it probably is too good to be true. Does it promise to cure a hitherto-incurable disease? Does it promise you can eat all you want and still lose weight? Does it promise there are no side effects of any kind? Does it remind you of the spiel of a used car salesman or a TV infomercial? Caveat emptor.
Who disagrees and why?
This is the most important question you can ask. It is rare for 100% of people to agree on anything. If you can find someone who disagrees, you can examine the reasons given for both opinions, and it will usually become obvious which side makes more sense. If you can’t find anyone who disagrees, it might be because it’s too new or because no scientist has taken it seriously enough to bother writing about it. In that case, withhold judgment and keep checking until someone does disagree.
Hope springs eternal, but true hope is better than false hope
If you are a desperate patient, it’s only natural to grasp at any straw of hope; but when the evidence is insufficient, the reasonable approach is to withhold judgment and wait for better evidence. You might think, “If it works, I don’t want to wait” but history teaches us that the great majority of these things don’t pan out. It might not do any harm, but then again it might; there might be adverse effects that haven’t been identified yet, it might raise false hopes only to dash them, and if nothing else it might waste time and money or interfere with getting more appropriate care. When you take an inadequately-tested medicine, you are essentially offering yourself as a guinea pig in a haphazard uncontrolled experiment that doesn’t even keep records. Of course, that’s your privilege; but I hope you would do it with your eyes open, with a realistic understanding of the state of the evidence.
Note: We sometimes take requests, as I did for today’s post. If you have done your best to apply these suggestions and are still uncertain whether the claims for a treatment are valid, drop us a line.

   Cognitive Traps
Harriet Hall 
Link
In my recent review of Peter Palmieri’s book Suffer the Children I said I would later try to cover some of the many other important issues he brings up. One of the themes in the book is the process of critical thinking and the various cognitive traps doctors fall into. I will address some of them here. This is not meant to be systematic or comprehensive, but rather a miscellany of things to think about. Some of these overlap.
Diagnostic fetishes
Everything is attributed to a pet diagnosis. Palmieri gives the example of a colleague of his who thinks everything from septic shock to behavior disorders are due to low levels of HDL, which he treats with high doses of niacin. There is a tendency to widen the criteria so that any collection of symptoms can be seen as evidence of the condition. If the hole is big enough, pegs of any shape will fit through. Some doctors attribute everything to food allergies, depression, environmental sensitivities, hormone imbalances, and other favorite diagnoses. CAM is notorious for claiming to have found the one true cause of all disease (subluxations, an imbalance of qi, etc.).
Favorite treatment.
One of his partners put dozens of infants on Cisapride to treat the spitting up that most normal babies do. Even after the manufacturer sent out a warning letter about babies who had died from irregular heart rhythms, she continued using it. Eventually the drug was recalled.
Another colleague prescribed cholestyramine for every patient with diarrhea: not only ineffective but highly illogical.
When I was an intern on the Internal Medicine rotation, the attending physician noticed one day that every single patient on our service was getting guaifenesin. We thought we had ordered it for valid reasons, but I doubt whether everyone benefited from it.
Recognizing warblers.
Like birdwatchers, hospitalists like Palmieri learn to identify which doctor admitted a patient. Child doesn’t appear sick; admitting diagnosis is “occult bacteremia”; patient was given an intramuscular injection of Cephotaxime in the office — oh, that must be Dr. X.
Rapid identification vs pareidolia
Humans are good at pattern recognition. This allows experienced clinicians to make rapid diagnoses, but it also allows us to see the Virgin Mary on a grilled cheese sandwich.
Rooster syndrome
Rooster crows, sun comes up; therefore rooster made sun come up. Baby had colic, was given treatment X, colic resolved; therefore X cures colic. In reality, colic resolves spontaneously by 3-4 months of age and X was useless.
Copycats
Mimicking what other physicians in the community are doing.
Availability
Choosing a drug because you have samples handy that the drug rep left.
Ulysses syndrome
Ulysses went from one adventure to another in the odyssey of returning home from the Trojan War. A false positive test can lead to a fruitless odyssey of further investigation: tests lead to more tests, maybe even invasive procedures and harm to the patient. Eventually it is realized that the patient has been healthy all along.
Unnecessary lab tests
Sometimes tests are done in a scattershot attempt to find something, anything. Palmieri’s pathologist wife directs a laboratory and frequently gets calls from doctors who have ordered an unfamiliar test and have no idea what to do when they get an abnormal result. Instead of getting an individual chemistry test, we get SMAC panels because the machine is there and it’s so convenient and cheap. With 20 tests on these panels, there is a 66% probability that at least one test will be outside normal limits on a perfectly healthy normal person.
Defensive medicine
With the present legal climate, doctors sometimes do tests or treatments with an eye to how things would look in court, rather than for the direct benefit of the patient.
Showmanship
Ordering tests to impress the patient that the doctor is being thorough and is actually doing something.
Hardwired fallibility
Our brains do not function in a rational, objective, logical way. We have built-in psychological mechanisms and defects in information processing; our brains have evolved a repertoire of tricks and shortcuts that serve us well in everyday life but that must be overcome for critical thinking and science.
Confirmation bias
Once we form a belief, we seek out evidence that confirms it and reject evidence that contradicts it. We are all biased, but by being aware of our biases we can activate a self-correcting mechanism.
Over-generalization
We form opinions about the many based on our experience of a few. We may base our idea of a disease on a patient who had an atypical presentation, or tend to avoid using a drug because of a patient who had an uncommon side effect. Radiologists who have missed a diagnosis are tempted to over-interpret x-ray findings for a time afterwards.
Anchoring
We tend to reach an early diagnosis and cling to it even when subsequent evidence doesn’t fit. We tend to accept the diagnosis of the referring physician rather than going back to square one to make up our own mind.
Diagnosis momentum
An early possibility becomes a presumptive diagnosis and gains legitimacy as it is repeated by more and more health care providers.
Framing
We seek a diagnosis within the context of how the information is presented to us. Palmieri tells about a boy who presented with “frequent throat infections.” He was referred to ENT and even had a tonsillectomy before it was discovered that he had never even had a sore throat, only unexplained fevers that had been falsely attributed to throat infections but that eventually turned out to be due to juvenile rheumatoid arthritis.
Miscommunication and assumptions
Palmieri describes a case where an ENT consultant was called in directly by the worried parents of a child hospitalized with an ear infection. He assumed that they and the pediatrician must have wanted him to put in PE tubes; otherwise there would have been no earthly reason for a consult. He had booked an OR and scheduled the patient for surgery before it became clear that the child had a first ear infection that was responding to treatment, that ENT input was unnecessary, and that PE tubes were clearly not indicated.
Algorithms
We simplify our approach to complex problems by following algorithms like “if the white count is over 15,000, give antibiotics.” This is not always appropriate. Algorithms provide a convenient framework, not an unalterable directive.
Tunnel vision
We are cautioned against thinking of zebras every time we hear hoofbeats, but we often fall into the opposite trap: we tend to fixate on the diagnoses we commonly see in our practice and not consider rare possibilities. On a recent episode of the television show “Untold Stories from the ER” there was a toddler who was refusing to walk because of leg pain. They took x-rays looking for fractures to confirm their initial diagnosis of child abuse. It turned out he had scurvy, a vitamin C deficiency that simply doesn’t occur in the 21st century US — but it did, because he was refusing all foods but oatmeal and his uneducated parents didn’t know there was anything wrong with catering to his wishes.
Conclusion
In medical school, doctors learn science but they may not learn to think like a scientist. Once out in practice, they become vulnerable to unproven claims, myths, and pseudoscience; and they are encouraged to give advice based on common sense and intuition rather than on evidence. Not just doctors but everyone needs to better understand the cognitive traps we all fall into. Since our human brains are inherently fallible, only critical thinking and good science can keep us on track. A major theme of this blog is that good science is essential for correcting our errors.

HIV Denial and “Just Asking Questions”
Steven Novella on February 12, 2014
Link
The “just asking questions” maneuver is familiar to many skeptics. The idea is to feign neutrality, to insulate oneself from accountability or being held to answer for any specific position, but meanwhile to sow doubt about a scientific claim by raising (dubious) questions.
Sometimes the “I’m just asking questions” gambit also tries to disguise itself as sincere journalism. That’s what journalists do, right, ask the tough questions, uncover the uncomfortable truth?
I find this approach particularly deceptive. It tries to hide the fact that the journalist is working off of a particular narrative. Asking questions is, in fact, just another narrative style, one that is meant to lead the reader/viewer to a particular conclusion about the subject. The narrative determines what questions are asked and how they are answered.
A perfect example of this deceptive approach is the HIV denialist movie, House of Numbers. Here is the synopsis from the movie’s website:
What is HIV? What is AIDS? What is being done to cure it? These questions sent Canadian filmmaker Brent Leung on a worldwide journey, from the highest echelons of the medical research establishment to the slums of South Africa, where death and disease are the order of the day. In this up-to-the-minute documentary, he observes that although AIDS has been front-page news for over 29 years, it is barely understood. Despite the great effort, time, and money spent, no cure is in sight.
Hyping ignorance is one denialist strategy. Leung asks questions, pretending to be neutral, but implying that those questions are controversial or unanswered. He characterizes AIDS as “barely understood” and summarizes the current medical situation as “no cure in sight.” The summary continues:
The HIV/AIDS story is being rewritten, and this is the first film to present the uncensored POVs of virtually all the major players — in their own settings, in their own words. It rocks the foundation upon which all conventional wisdom regarding HIV/AIDS is based. If, as South African health advocate Pephsile Maseko remarks, “this is the beginning of a war…a war to reclaim our health,” then House of Numbers could well be the opening salvo in the battle to bring sanity and clarity to an epidemic clearly gone awry.
Clearly, this is a denialist movie. Rocking the foundation of conventional wisdom, bringing clarity, and being uncensored are all standard parts of the denialist rhetoric.
A brief history of HIV
HIV denial was galling in the 1990s, but it is even more so today given the stunning advances in HIV medicine.
On June 5th 1981 the CDC published a report of five cases of a rare pneumonia (PCP pneumonia) in five otherwise-healthy young gay men. In 1982 it was recognized that these and other patients were suffering from an immune deficiency syndrome and opportunistic infections. In 1983 a retrovirus was discovered as the probable cause of this new syndrome (AIDS). The virus was initially called Lymphadenopathy Associated Virus but later renamed Human Immunodeficiency Virus (HIV). In 1985 the first blood test for HIV was approved. In 1987 the FDA approved the first drug for the treatment and prevention of HIV – AZT.
Over the next 20 years scientists developed what is called highly active antiretroviral therapy, or HAART, treatment. This is a cocktail of multiple drugs designed to interfere with HIV at multiple steps in its infection and replication. As these drugs continue to be developed, the life expectancy of those with HIV increases. Just last year, in 2013, data suggests that those with HIV who start HAART treatment early have essentially a normal life expectancy. In 30 years HIV went from a short term death sentence to a chronic illness with an essentially normal life expectancy.
There have been only a few “cures” of HIV, meaning that those with a documented infection are now virus free after treatment. One such cure was effected with a bone marrow transplant, and another with high dose HAART in an infant infected at birth. These approaches will likely not be applicable to most patients with HIV.
Research into an HIV vaccine is ongoing. The best we have done so far is a phase III trial showing 31% efficacy in preventing initial infection. This is a modest result, but shows we are making progress. HIV is particularly challenging for various reasons – it mutates throughout the infection, it evades the immune system, and it can hide in a dormant state.
In short, HIV is one of the stunning success stories of modern medicine. Science rapidly discovered the nature and cause of AIDS, we have had a steady increase in our understanding of this virus, it is now a manageable disease and we continue to make progress toward a vaccine and even possible cure.
HIV denialism
Throughout this 30 year success story, there have been those who have denied that HIV exists and/or is the cause of AIDS. Initially they claimed that the lack of clear benefits from anti-retroviral therapy was evidence that the HIV hypothesis was wrong. If that were true, then the stunning success of HAART should be considered evidence that the HIV hypothesis is correct. In fact, the success of HAART has taken the wind out of the sails of much HIV denialism, and yet it limps on.
In House of Numbers, Leung (who is just asking questions) asks: Is there really a consensus on HIV? Is the test reliable? Why do doctors ask about risk factors? Why doesn’t everyone who is exposed get the disease? Why is the disease so variable?
And, of course, the Big Pharma gambit – isn’t HIV just an invented fiction so that pharmaceutical companies can make money from selling drugs?
Jeanne Bergman over at AIDSTruth.org does a good job of answering these questions. Yes, there is a robust consensus on HIV, based upon mountains of evidence. Yes, the test is reliable. Asking about risk factors is standard practice. Not everyone exposed to an infectious illness catches the disease. The disease is actually very consistent. The opportunistic infections may vary by region, however.
Regarding the Big Pharma gambit – I suspect that someone living a normal life with a normal life expectancy on HAART medication doesn’t think the pharmaceutical industry is ripping them off. If you search for HIV on Pubmed you get 265,637 results. That is a lot of research. In there are clinical trials and systematic reviews of HAART therapy demonstrating clear efficacy. There is no controversy over the appropriateness and effectiveness of HAART therapy in those with HIV.
The response to Bergman’s deconstruction of House of Numbers by the denialists is predictable. They simply double down on the “we’re just asking questions” gambit. On the HoN Website they include this formal response by the HIV denialist Perth group:
The “legitimate scientist” and Nobel laureate Jacques Monod: “In science, self-satisfaction is death. Personal self-satisfaction is the death of the scientist. Collective self-satisfaction is the death of the research. It is restlessness, anxiety, dissatisfaction, agony of mind that nourish science”.
Perter Abelard: The first key to wisdom is assiduous and frequent questioning…For by doubting we come to inquiry, and by inquiry we arrive at truth”.
They further state:
Howard Temin, the father of modern retrovirology: “when an experiment is challenged no matter who it is challenged by, it’s your responsibility to check. That is an ironclad rule of science, that when you publish something you are responsible for it…even the most senior professor, if challenged by the lowliest technician or graduate student, is required to treat them seriously and consider their criticisms. It is one of the most fundamental aspects of science” (emphasis in original).
HIV scientists do not respond to challenges. In fact, according to the rules at AIDSTruth, “We will not engage in any public or private debate with AIDS denialists or respond to requests from journalists who overtly support AIDS denialist causes”. Apparently these “legitimate scientists” cannot stick to their own rules.
Here we see the true agenda of “just asking questions” laid bare. Denialists are not asking sincere questions with the goal of further exploration and getting closer to the truth. They have an agenda of casting doubt on established science. They will tirelessly ask questions in order to create the false impression of controversy where none exists, all the while pretending to be skeptics who just value doubt as a scientific tool.
The quote above from Temin does not apply. Even the lowliest technician who is asking questions is assumed by Temin to be sincere in their motives. This does not apply to denialists, who abuse questioning, abuse skepticism, and abuse doubt for an a priori agenda of denial.
“Just asking questions” is a win-win for denialists. If you respond, you validate the notion that there is a controversy, a question that needs answering. If you refuse to engage, then you are hiding something and not being a true scientist.
Conclusion
The medical science behind HIV/AIDS is astoundingly robust. Of course there is much more to be discovered, and we need to further develop HIV treatments. An effective vaccine and/or effective cure are greatly needed, especially if we are ever going to eradicate HIV. Eradication is possible, however, as HIV has no significant non-human reservoir.
Denialists, however, confuse consensus with being closed minded. They confuse refusal to engage with cranks with being evasive. They pretend to be just asking questions when in reality they clearly demonstrate that they are not honest brokers. Rather they are merchants of doubt and confusion with an agenda.
I don’t know if Leung came to this question a denialist, or if he is just a journalist who decided to build his career off a “man bites dog” story. A movie confirming the standard scientific view of HIV would not have been as sensational.
We also have to acknowledge the potential harm of HIV denialism. Trying to convince the public that they are not being told the true story of HIV has the potential to convince some people not to trust the system, and therefore not to get tested or take HAART medication. With serious medical issues, “just asking questions” has a body count attached to it.

   A Budget of Anecdotes
Mark Crislip
Link
Anecdotal evidence. An oxymoron? Or a valid approach to understanding data?
The problem is there are different kinds of anecdotes, used for different purposes, but the purpose of anecdotes is rarely if ever defined explicitly. Anecdotes are used for one purpose by one speaker/writer but interpreted in a different context by the listener/reader. People love anecdotes, especially if the anecdotes are about them or their beliefs. Anecdotes are how patients transmit the particulars of their disease to their health care providers. The medical history, as taken from the patient, is an extended anecdote, from which the particulars of the disease have to be extracted. Anecdotes are how physicians explain disease and treatments. Anecdotes are a tool with which teachers instruct their students. Anecdotes are how CAM proponents validate their particular system, and how skeptics invalidate them.
Anecdotes are useful tools for presenting yourself and your ideas. The convention season is over and is was striking how the candidates attempted to win over voters with anecdotes about their lives rather than the particulars of their policies. Using variations of ‘anecdote’ as a pubmed search term yields little of substance. The predominant theme on medline is to contrast anecdotes with evidence, always to the detriment of anecdotes. Anecdotes have power to influence far greater than evidence.
On The Skeptics Guide to the Universe #165 there was an interview with Ben Goldacre, who noted that there was the popular misbelief that the MMR vaccine was a cause of autism. The belief waned not when the voluminous data on the safety and lack of association with autism and the MMR was released, but when it was discovered that the primary proponent of the MMR/autism link received large sums of money to testify about that MMR/autism link. It was the anecdote about his conflict of interest that invalidated the idea, not the science.
The participants of the comments section use and deride anecdotes with equal frequency. I cannot find that a classification of anecdotes has been attempted before, so I am sure what follows is incomplete. I am equally sure the readers of the blog will increase my budget (1). This is a catalogue of anecdotes as they are used in medicine, by clinicians and patients and in discussions of CAM. With, of course, anecdotes.
1) Anecdotes used to describe new observations.
Occasionally an astute clinician notices something new, unexpected, or different in a disease manifestation or a side effect of treatment. If he has the time, it gets published or reported. Otherwise it gets filed away in the back of the brain and next time a similar case is seen, the oddities of the prior case may be remembered and mentioned to others. An anecdote may have a kernel of truth that will be validated later, or be forgotten, or be disproved. I remember as a resident I would comment that if a patient smoked three packs of cigarettes a day, they must be schizophrenic. At the time it was somewhat of a smart ass comment, but later it was shown that smoking helps ameliorate the symptoms of schizophrenia. I need to pay closer attention to my own anecdotes.
Homeopathic remedies are entirely based on anecdotes used to describe new observations, but as practiced on bizarro world. Each remedy has a ‘proving’ (2) where healthy people are given a homeopathic concoction and watched for every symptom, sometimes for days afterwards, which are recorded carefully and used to determine the use of the homeopathic preparation. Each homeopathic proving is an anecdote used as a new observation as to what the homeopathic remedy is to be used for.
2) Anecdotes used as archetype
Diseases, as described in textbooks, can be dry as dust. The medical literature, so called, resembles literature much as the dictionary. The style of medical writing may be maximally efficient for imparting knowledge, but it is also maximally dull. Adjectives and adverbs are frowned upon, and it is the rare article or textbook that has a clever phrase or bon mot. Lists of symptoms and the percentages of their occurrences make up the standard medical description of disease.
Unusual diseases are hard to diagnose if you have never seen it before. Knowing about a disease from reading a textbook does not mean you are likely to recognize it the first time you see it. It is good to have a prior case, an archetype, to which you can compare both the textbook descriptions and the subsequent potential cases.
Take Still’s Disease.
I know the presentation of Still’s: fever, rash, myalgias and arthralgias with sore throat, swollen lymph nodes and hepatosplenomagaly. Not really all that specific in its presentation. This description can also be applied to mononucleosis, some cancers, rheumatic fever and a few other diseases. As mentioned before, the body has a limited number of ways of responding to illness.
The first time I saw a Still’s was in clinic on a Friday afternoon, I had no idea what the patient had. I only knew she looked sick and her labs suggested something very bad was going on. So I admitted her to the hospital and left for the weekend. When I returned on Monday she had been discharged and the hospitalist told me, oh yeah, it was Still’s “like oh yeah, it is raining out”. She was a textbook case, but a textbook never does justice to the patient. Now when I see a fever and I am concerned about Still’s, I think about that particular case and compare her to the case at hand. When I discuss Still’s with the residents or read about Still’s, I use as an archetype that first case as an anchor for understanding.
In medicine, these archetypes are based on the best scientific understanding of disease. When you understand the pathophysiology of Still’s, or Cat Scratch, or any disease and are confronted with a patient who may have a given disease, it is good to have an archetype to which you can refer and apply the pathophysiologic understanding. It puts a face on the disease.
3) Anecdotes used as an example of a general case
This has a long tradition in medicine. I need to give a lecture about Zoonosis (animal related infections). Often the lecture will start with a case presentation. It may be a real case, or it may be a combination of several cases. The purpose of the anecdote is to tie the lecture, which is often filled with the dry results of clinical studies, to a real person with a real disease. Unlike the archetype, which is used to help understand unusual diseases, the anecdote is used to describe common medical conditions and is codified in the case presentation. Anecdotes are effective tools to aid learning about a common disease.
4) Anecdotes used as an example of atypical manifestations of common disease.
We all get blindsided by common diseases that present oddly. Classic angina presents with pain that radiates into the neck and down the left arm, often brought on by exercise. As a resident I had a patient whose angina was left wrist pain with exercise. I like to tell that anecdote to residents to emphasize the need to be on the lookout for atypical manifestations of common diseases. Medicine would be so much simpler if patients would read the textbooks before coming to the hospital.
Corollaries include anecdotes used as examples of typical manifestations of uncommon disease (a cousin of the archetype anecdote) and the anecdote used as examples uncommon manifestations of uncommon diseases. All of these disease manifestations are best remembered through anecdotes, and serve as cautionary tales. If you are not careful, you too may be fooled and miss an important disease.
5) Anecdotes used as an example of proven concepts
Syphilis has a wide variety of manifestations and is best treated with penicillin. Proven concepts. When I teach a resident or discuss a case, I may use an anecdote about one of the many syphilitics I have treated over the years.
Homeopathy cannot work for any disease except thirst, since all homeopathic concoctions contain nothing but water. A proven concept in so far as it is possible to prove a negative. I could present a case where a patient used a homeopathic remedy for syphilis and the disease progressed unhindered by the homeopathic concoction.
Both the efficacy of penicillin for the treatment of syphilis and the lack of efficacy of homeopathy are proven concepts, illustrated by anecdotes.
6) Anecdotes used as an example to prove an unproven concept
This would be an example of the justly derided anecdotal evidence, and differs from the other, perhaps more benign, forms of anecdotes.
Homeopathy does work, here is the proof: I gave a patient with joint pain a homeopathic concoction and the patient improved. The knee pain went away. Proof of the efficacy is in the successful intervention.
It is here that the motto “the plural of anecdote is anecdotes not data” is most applicable. It is whether the anecdote is being used to prove a concept, or used as an example of a proven concept is where practitioners of CAM and real medicine differ, both on how the anecdote should be used and the validity of the anecdotes use.
It is the use of anecdotes as a proof of concept upon which the basis of much of CAM rests, and is the most unreliable use of anecdotes. Given the vagaries of human memory, the difficulty in proving causality, and the lack of biologic and physical possibility of most CAM interventions, using anecdotes as proof of CAM effectiveness is a particularly problematic use of anecdotes. Even if Dr. Weil tarts them up by calling them uncontrolled clinical observations.
Saying “the plural of an uncontrolled clinical observation is uncontrolled clinical observations, not data” just doesn’t seem as catchy.
It is this kind of anecdote that has the most power, both in CAM and in the reality-based practice of medicine. An anecdote where something happened has far more power to convince than an anecdote where nothing happens. A patient has an adverse reaction after a vaccine or gets better after taking echinacea. The anecdote suggests causality. If you respond that clinical studies demonstrate the lack of data to support their anecdote, they will look at you like a dorky brainiac divorced from the real world. If you counter with an anecdote where, under the same circumstances, for you nothing happened after the vaccine or the echinacea, you will probably get the same “man what a clueless dork” roll of the eyes. Hits have far more power than misses, your anecdote will be dismissed, and you will be branded a hypocrite since in the past you have called anecdotes worthless.
Skeptics of CAM are doomed to forever lose against the formidable power of the positive anecdote, as one positive anecdote is worth 1000 studies and 10,000 negative anecdotes.
Can anecdotes ever be considered proof? There may be times when anecdotes can be definitive without the need for classical clinical trials (3). Some types of adverse drug reactions may be considered proven with anecdotes. In “Anecdotes that provide definitive evidence” the authors provide 4 instances that they feel anecdotes are sufficient data to prove a hitherto unproven idea.
1) Extracellular or intracellular tissue deposition of the drug or a metabolite
2) Specific anatomical location or pattern of injury
3) Physiological dysfunction or direct tissue damage that can be proved by physicochemical testing
4) Infection as a result of administration of a potentially infective agent or because of demonstrable contamination.
These are all very narrow instances of anecdotes as proof, and may be unreliable, as case reports for adverse drug reactions are rarely confirmed in subsequent studies (4).
I would consider a stroke immediately following cervical spine manipulation in the same category where the anecdote is convincing. I am sure others will disagree or have other anecdotes they consider definitive. What level of plausibility where an anecdote could be considered sufficient proof to confirm a hypothesis is uncertain, depending on prior plausibility and will vary from person to person.
But as proof of a concept in CAM, in most instances one has to combine aphorisms: “the plural of anecdote is anecdotes, not data” and “extraordinary claims require extraordinary evidence.”
6) Anecdotes used to guide treatment when there is no data.
Grand Rounds this week was on Pulmonary Hypertension, a rare disease with an array of new, effective, and very expensive therapies. Pulmonary hypertension has a variety of causes and stages. When do you start therapies and in what order. There is some data, but given the rarity of the disease, there are many instances where the particular case does not match the known information. So as a clinician you rely on the anecdotes of others and your past experience to guide therapy.
The same issue occurs in AIDS treatment. There are 28 different HIV medications and more coming. There is some data to suggest the best initial therapy, but as resistance and intolerance occurs to the medications, often it is anecdotal data that determines which medications are used.
Clinicians get comfortable with one therapy and perhaps had poor outcomes with other therapies, and these anecdotes influence further interventions. Unlike CAM, there is biologic plausibility behind the anecdotes for your interventions. You always understand that using anecdotes to help decision making is a temporizing measure because, and again unlike CAM, there will be future studies that will help guide optimal therapy.
7) Anecdotes used as the history of the present illness
People love stories. They love to tell stories, especially about themselves and about their diseases. Patients do not give organized medical histories. They tell extended anecdotes about their illness. What a patient tells me about their disease and what really happened are often two different issues and all too often (especially when it is the last consult on a Friday) the patient is particularly loquacious with the details of their disease that have no bearing of the diagnosis. Part of being a doctor is learning to extract the pertinent information from the sometimes endless detail in the anecdotes.
8) Anecdotes used to provide understanding of the unknown.
Often the diagnosis and treatment of a disease can be uncertain, and by referring to themselves and others with anecdotes about the process, it puts the illness in a context that renders it explainable to the patient and their close friends and family. Patients frequently ask my experience with their disease, and I used anecdotes about past successes and potential pitfalls to explain and personalize their disease. The anecdotal experiences are, I think, more reassuring and understandable, than just recitation of the statistics of their disease, although I provide that as well. Patients don’t care that 98% of people with streptococcal endocarditis are cure, since for them it is all or nothing. So I mention the success rate, the outcomes, and allude to the experiences I have had with other patients.
9) Anecdotes used to guide diagnosis or therapy because of either prior unusual outcomes, good or bad.
This is the hard one, as a clinician, to be conscious of. If you have ever had a particularly horrible outcome from the care of a patient, you will remember that case. Vividly and forever. When you have a similar case, that anecdote will float though your head and may alter your approach, perhaps inappropriately so. Bad outcomes and complications have enormously greater impact on care than good outcomes, and I bet most of us made rounds with the ghosts of patients past, who look over our shoulders and guide our therapy. These anecdotal ghosts can exert more power than evidence, especially when the data is sparse or the outcome was particularly bad.
It is rare for anecdotes to be used in a single context. Language is not precise and anecdotes can have many interpretations. Anecdotes are flexible and one anecdote can have many functions to the user and many interpretations by the listener. Perhaps we should be more precise in our use of anecdotes. When professional societies make recommendations, the strength of the data has a rating system, of which there are a variety (5).
Perhaps, and I am only half serious when I suggest this, we need a similar classification for anecdotes when we discuss medicine and, especially, CAM.
Someone tells an anecdote about how CAM works or how it fails. How is it being used? After the anecdote, put the type of anecdote in a parenthesis, just like they big boys do when making recommendations. Then we can assess the validity of the anecdote in the context the writer intended, rather than the context applied by the reader.
Anecdotal Types
Type 1) Anecdotes used to describe new observations.
Type 2) Anecdotes used as archetype.
Type 3) Anecdotes used as an example of a general case.
Type 4a) Anecdotes used as an example of atypical manifestations of common disease.
Type 4b) Anecdotes used as examples of typical manifestations of uncommon disease.
Type 4c) Anecdote used as examples uncommon manifestations of uncommon diseases.
Type 5) Anecdotes used as an example of a proven concepts.
Type 6) Anecdotes used as an example to prove an unproven concept
Type 7) Anecdotes used as the history of the present illness.
Type 8) Anecdotes used to provide understanding of the unknown.
Type 9a) Anecdotes used to guide diagnosis or therapy because of either prior unusual outcomes, good.
Type 9b) Anecdotes used to guide diagnosis or therapy because of either prior unusual outcomes, bad.
I have an anecdote that suggests such an approach will work (Anecdote Type 1).
—————————
References etc
1) Budget: archaic a quantity of material, typically that which is written or printed. An old classic is a Budget of Paradoxes.
2) Like complementary, which complements nothing, and alternative, which isn’t, a ‘proving’ proves nothing.
3) Anecdotes that provide definitive evidence 2006;333;1267-1269 BMJ . Jeffrey K Aronson and Manfred Hauben
4) BMJ. 2006 Feb 11;332(7537):335-9. Case reports of suspected adverse drug reactions–systematic literature survey of follow-up.
5) BMC Med Res Methodol. 2006; 6: 52. A system for rating the stability and strength of medical evidence

   The Role of Anecdotes in Science-Based Medicine
Steven Novella
Link
While attending a lecture by a naturopath at my institution I had the opportunity to ask the following question: given the extreme scientific implausibility of homeopathy, and the overall negative clinical evidence, why do you continue to prescribe homeopathic remedies? The answer, as much as my question, exposed a core difference between scientific and sectarian health care providers. She said, “Because I have seen it work in my practice.”
There it is. She and many other practitioners of dubious modalities are compelled by anecdotal experience while I am not.
An anecdote is a story – in the context of medicine it often relates to an individual’s experience with their disease or symptoms and their efforts to treat it. People generally find anecdotes highly compelling, while scientists are deeply suspicious of anecdotes. We are fond of saying that the plural of anecdote is anecdotes, not data. Why is this?
Humans are social storytelling animals – we instinctively learn by the experience of others. My friend ate that plant with the bright red berries and then became very ill – lesson: don’t eat from that plant. This is a type of heuristic, a mental shortcut that humans evolved in order to make quick and mostly accurate judgments about their environment. From an evolutionary point of view it is probably statistically advantageous just to avoid the plant with the red berries rather than conduct blinded experiments to see if it really was the plant that made your friend sick.
Further, the most compelling stories are our own. When we believe we have experienced something directly, it is difficult to impossible to convince us otherwise. It’s just the way humans are hardwired.
Understanding the world through stories was a good strategy in the environment of our evolutionary history but is far too flawed to deal with the complex world we live in today. In fact, the discipline of science developed as a tool to go beyond the efficient but flawed techniques we evolved. Perhaps, for example, your friend became ill because of the raw eggs he consumed earlier in the day, and the plant had nothing to do with it. Evolutionary pressures favored a more simplistic approach to nature, one that tended to assume that apparent patterns were real.
In today’s modern society we are confronted with a dizzying array of apparent patterns and using the simple rules of thumb we evolved to deal with them is not adequate. Whether or not a treatment works for a symptom or disease is a good example. Symptoms tend to vary over time, some may spontaneously remit, and our perceptions of symptoms are susceptible to a host of psychological factors. There are also numerous biological factors that may have an effect. If we are to make reliable decisions about the effects of specific interventions on symptoms and diseases we will need to do better than uncontrolled observation, or anecdotes.
The primary weakness of anecdotes as evidence is that they are not controlled. This opens them up to many hidden variables that could potentially affect the results. We therefore cannot make any reliable assumptions about which variable (for example a specific treatment) was responsible for any apparent improvement.
Here are some specific factors that make it difficult to impossible to reliably interpret anecdotal medical evidence:
Regression to the mean: This is a statistical phenomenon whereby any extreme variation is likely to be followed by a more average variation – by chance alone. Many diseases have variable or fluctuating symptoms – good days and bad days, or periods of exacerbation followed by periods of relative relief. If a person seeks out a treatment when their symptoms are severe, by chance alone this is likely to be followed by a period when the symptoms are not as severe.
Most illnesses are self-limiting: The old saying goes that if you don’t treat a cold it will last for seven days, and if you treat it it will last for a week. Most ailments get better or improve on their own, therefore most treatments will be followed by symptom resolution even if the treatment has no biological effect. More broadly, all illnesses have a natural history, a course they typically follow over time. In order to know if a treatment is affecting that course it has to be compared to patients who are not treated, or receive a different treatment.
Multiple treatments: Often people will try multiple treatments for a disease or ailment making it impossible to tell which treatment had a beneficial effect, if any. Multiple treatments may be taken all at once, or sequentially. For example, a person with a long term illness (but one destined to have a period of relative relief) tries treatment A without effect, then treatment B without effect, then treatment C which is followed by improvements in their symptoms. They then credit treatment C, recounting how multiple other treatments had failed. However, since the person was trying some new treatment most of the time at any point that their symptoms improved there would be a treatment they could credit with that improvement.
Dead men tell no tales (the problem of reporting bias): Cancer survivor groups do not contain people who died of their cancer. Those who die of a disease are not around to give their anecdotes. There is therefore a built in reporting bias. Also, those who feel they were helped by a treatment are much more likely to boast about it than those for whom there was no apparent benefit. People like to tell the tale of the miracle cure they found and had faith in, despite the skeptics and naysayers – but their vision paid off as the treatment worked for them. People have no motivation to recount their experience with the novel treatment that did not work. Further, patients who feel they are being helped by their doctor or practitioner are more likely to return. Those who feel the treatments are not working may not come back at all to report the treatment failure.
Confirmation bias: It is a well-described psychological phenomenon that we tend to seek out and remember information that confirmed what we already believe, or want to believe, and we avoid, forget, or explain away disconfirming evidence.
Vague outcome measures: Good clinical trials use objective outcome measures – those that are binary (like death or survival), quantitative (like a blood level), or are based upon a specific physical finding. Subjective symptoms do not make good outcome measures because they require that judgments be made, and that introduces yet another variable. Should you count those mild sniffles as having a cold? If you are taking a remedy that you think will help you avoid colds you may dismiss those sniffles and report (and even remember) that you did not get any colds while taking the treatment.
The Placebo Effect: The placebo effect is actually a host of many effects that give the appearance of a response to an inactive treatment. These factors include many of the things I listed above, but also other variables that may alter health outcomes or symptoms. See here for a more complete discussion.
The Fallibility of Human Memory: Medical students quickly learn that one of the biggest challenges in taking a medical history is that people are poor historians, which a polite way of saying that human memory is terrible. Anecdotes largely depend upon an individual’s memory of their illness and treatment. This introduces many new variables. There is, for example, a tendency for people to conflate different events in their memory into a single event, or to combine details from various events. There is also a tendency for details to evolve over time to make a story more clean and profound. So people may, in their memory, exaggerate the severity of their symptoms prior to treatment, exaggerate the response to the treatment, clean up the timeline of events so that improvement began very soon after a treatment (rather than before or long after), forget other treatments that were taken, distort what they were told by their various health care providers, etc. I have had countless opportunities to compare a patient’s memory of their illness and treatment to the documented medical records, and the correlation ranges from poor to completely wrong.
For these, and other reasons, scientists have learned not to trust anecdotal reports – or rather to have a realistic assessment of their reliability. This is why it always strikes me as profoundly naive when anyone presents anecdotal evidence as if it is compelling, or even argues that anecdotes should be relied upon as valid evidence.
We also have history to inform our opinions about anecdotes. Western practitioners relied upon the humoral theory of health and illness for thousands of years. Apparently thousands of years of anecdotal experience did not inform them that their treatments were worthless or harmful. Dr. Abrams became wealthy by selling a machine to diagnosis and treat ailments. His devices were widely used, with millions of people swearing by their effectiveness. It worked for them, and their experience was unshakable. When Abrams died it was discovered that his machines (previously protected from inspection) were filled with useless random machine parts. At the turn of the century radioactive tonics were popular, until prominent proponents began seeing the ill effects of radiation poisoning.
The point of these examples is that anecdotal evidence led many people to conclude that these interventions worked. They are useful examples because they are no longer accepted, humoral theory was replaced by scientific medicine, Abrams devices were dramatically exposed, and radiation therapy is directly harmful. But for treatments that are not directly harmful (and least not in an obvious way) or where there is no “man behind the curtain” to dramatically expose, all we have are the anecdotes – and clearly they are not reliable.
Even in mainstream medicine we have learned to distrust anecdotal evidence, even our own. The history of medicine is strewn with treatments that seemed to work but then were abandoned when scientific evidence showed otherwise. The classic example of this is mammary artery bypass for cardiac angina – it seemed anecdotally to work, but it didn’t.
But should anecdotes play any role in medical evidence? Yes, but a very minor and clearly defined one. Anecdotes, with all their weaknesses, are real life experience. It is possible that a treatment does in fact work and personal experience may be the first indication that there is a meaningful biological effect in play. But here are two limiting factors in how anecdotes should be incorporated into medical evidence:
The first is that anecdotes should be documented as carefully as possible. This is a common practice in scientific medicine, where anecdotes are called case reports (when reported individually) or a case series (when a few related anecdotes are reported). Case reports are anecdotal because they are retrospective and not controlled. But it can be helpful to relay a case where all the relevant information is carefully documented – the timeline of events, all treatments that were given, test results, exam findings, etc. This at least locks this information into place and prevents further distortion by memory. It also attempts to document as many confounding variables as possible.
The second criterion for the proper use of anecdotes in scientific medicine is that they should be thought of as preliminary only – as a means of pointing the way to future research. They should never be considered as definitive or compelling by themselves. Any findings or conclusions suggested by anecdotal case reports need to be later verified by controlled prospective clinical studies.
Understanding the nature and role of anecdotes is vital to bridging the gap between the proponents of science-based medicine and believers in dubious or sectarian health practices (as well as the public at large). In my experience it is often the final point of contention between these two camps.
It is interesting to note that the scientific community has long ago made up its collective mind about the weaknesses and role of anecdotes. Logic and the lessons of history speak very clearly on this issue. But there are forces at work today that want to turn back the clock on scientific progress – they want to bring back anecdotes as a reliable source of medical evidence, essentially returning to the pre-scientific era of medicine. In some cases this is done out of frustration – that controlled scientific data has not validated a prior strongly held belief. In other cases it seems to be a calculated attempt to lower the bar of evidence to admit treatments that have not been validated by solid scientific evidence. In either case, this is not in the best interest of the health of the public.

Storytelling in Medicine
Harriet Hall on December 18, 2012
Link
We can’t stress often enough that anecdotes are not reliable evidence; but on the other hand, patient stories can serve a valuable purpose in medical education. Hearing how a disease affected an individual patient is more powerful than reading a list of symptoms in a textbook and is far more likely to fix the disease in the student’s memory. When I think of Parkinson’s disease, the first thing that comes to mind is my first patient with Parkinson’s and how he responded to levodopa; and the first thing that may come to many people’s minds is Michael J. Fox. Of course, we must realize that they may not be typical examples; but putting a face to a diagnosis serves as a memory aid and a hook to hang the rest of our knowledge on.
In his new book, The Power of Patient Stories: Learning Moments in Medicine, Paul F. Griner, MD relates more than 50 stories that distill the wisdom he has developed over a 58-year career of practicing medicine and teaching young doctors. He describes them as “stories that provided a learning moment for me.” It’s interesting to see how much medicine has changed over his professional lifetime and yet how cases from the 50s and 60s are still highly relevant. Ethical dilemmas and lessons about medical practice come alive under his pen. Each story is followed by incisive questions and exercises that engage the reader and challenge him to think about the issues.
Is She a Virgin?
Instead of pontificating, Griner humbly questions his own judgment. Suspicious parents once demanded that he examine their 14-year-old daughter to see if she was still a virgin. He thought it was an invasion of the girl’s privacy, and he knew the physical exam was not a reliable guide to whether the girl was sexually active. He talked to the girl in private and chose not to examine her but instead to suggest ways to improve her communication with her parents. He then told the parents she was probably still a virgin. His anger may have clouded his judgment; he considers that maybe he should have examined the girl because she might have been pregnant, and he wonders how he might have handled the situation differently. Then he asks “what would you have done?”
Exposed
When Griner was president of his hospital, he had an embarrassing experience. A dermatologist had him undress for a full skin exam and gave him a gown that was torn where the buttons met at the shoulder so it kept slipping off. The next day he called a meeting of his management team and entered the conference room wearing that gown and nothing else. His stunt impressed them and gave them vivid insight into the vulnerability patients feel. Changes were made. They still talk about it.
The Walking Dead
A patient who had had a heart attack was revived with an emergency chest incision and manual heart massage. He woke up, got off the table, and walked across the room with the doctor’s hand still inside his open chest. Then he collapsed and died. Why did they open his chest? Because defibrillators hadn’t been invented yet and closed chest massage for cardiac arrest wasn’t even described until 2 years later.
A Solution Worthy of MacGyver
It’s easy to forget that we have not always had portable EKG machines, monitors, and defibrillators. In the days before those were available, Griner was at a cardiac patient’s bedside when the patient became unresponsive. He diagnosed cardiac arrest due to ventricular fibrillation. He stripped the wires from the bedside radio, attached them to two long needles, plunged the needles into the patient’s chest, and had the nurse repeatedly insert the radio plug into the wall socket; the patient quickly revived. Always the skeptical scientist, he now wonders if the patient really was in ventricular fibrillation, since he had not been able to verify his diagnosis with an EKG, and since modern experts question whether a 120 volt shock could reverse v. fib. And he asks us to consider the ethical and legal implications of performing an unprecedented treatment like this on a patient without informed consent.
Looking Without Seeing
He asked a patient with fever and chills if his teeth chattered. The patient grinned and said, “Let’s look in the drawer and find out.” His false teeth were in the drawer of the bedside table! Griner stresses the importance of good bedside skills of listening, observing, and examining; and deplores the increasing tendency of young doctors to pay more attention to test results than to patients. The great majority of diagnoses can be made on the basis of the patient’s history alone, and most of the rest are made on the physical examination. Tests should be used to confirm diagnoses, not to make them.
Issues Worth Discussing
Griner talks about how today’s rapid improvements in technology may be outpacing our ability to apply it wisely. Doctors are losing their bedside diagnostic skills; it’s easier to order a battery of tests and hope something turns up. Excessive testing increases costs and can even harm patients.
New regulations allow medical residents to get more sleep. He relates being so exhausted during his training that once when he couldn’t get an IV started, he threw all the IV equipment on the floor and wished the patient would die just so he could get some sleep. New policies may reduce the risk of errors, but it can also be argued that they reduce the quality of education.
Are we upholding the ethical values of truthfulness, justice, dignity, and confidentiality? He discusses the ethics of hastening death in a patient suffering inexorably from terminal cancer, and admits he once killed such a patient with an overdose of morphine. He illustrates the problem of allocating scarce resources with his experience of having to decide which patients to treat with the pitifully few respirators available during an influenza epidemic. He asks what doctors should do when they think a colleague is incompetent, and illustrates it with an experience of his own with a superior in the military. (I could really relate to that one! You can read about my own experience in my book.)
Doctors today are pressed for time and tend to look more at their computers than at their patients. We all stand to lose when there is less time for human interactions. Griner wants to put a more human face back on medicine.
He describes several innovative teaching methods. One I found particularly intriguing was a program where 4th year medical students shadow a patient from admission to discharge and record every instance where care could be improved, everything from waiting times and housekeeping issues to medication errors and unprofessional behavior.
When one considers the amazing progress of science in its relation to medicine during the last thirty years, and the enormous mass of scientific material which must be made available to the modern physician, it is not surprising that the schools have tended to concern themselves more and more with this phase of the educational problem.
Griner might have said that, but he didn’t. Francis Peabody said it in 1927 regarding students at Harvard Medical School. Plus ça change, plus c’est la même chose.
Conclusion
In this entertaining gem of a book, Dr. Griner raises important questions that are well worth grappling with, and he teaches valuable lessons. I wish every medical student and health care professional could read this book. And anyone who likes to read true stories about medicine will find the book fascinating.

Another Misguided Cancer Testimonial
Mike Shedlock wrote a post about how he beat prostate cancer. In doing so, he provides a typical example of how difficult the medical literature can be for a layperson to read, and where they can get things wrong.
Harriet Hall on January 27, 2015
Link


An economic analyst, Mike “Mish” Shedlock, wrote a blog post to describe how he beat prostate cancer. When laymen and patients write about cancer, they are likely to get some things wrong. Mish’s story is full of typical misunderstandings and misinterpretations.
He interpreted his experience in his own way and did his own research into the medical literature, something he was not qualified to do. Prostate cancer is a very complex subject, and understanding the implications of published studies for treating patients can be difficult even for experts. In typical Dunning-Kruger fashion, he rejected the advice of his doctors, thinking he could do better.
What his tests showed
Mish had been getting prostate-specific antigen (PSA) tests every couple of years. He believes this is a routine test recommended for men as they get older. It’s not that simple: the PSA test has its uses, but when used as a routine-screening test on the entire population of older males, it is likely to do more harm than good. The U.S. Preventive Services Task Force (USPSTF) says there is insufficient evidence to recommend routine screening. I’ve written before about the prostate screening controversy.
He thinks any reading under 3.0 (or 4.0 or 2.5) is normal. It’s not that simple: there are no defined normal levels for PSA. Low levels can be seen in patients with cancer, and high levels can be a sign of a number of other things, like inflammation or the benign prostate enlargement (BPH) that men frequently develop as they get older. Whatever the level, what is most significant is not the level itself, but a rise over time. After two readings in the 1-2 range, Mish had a reading of 6.65 followed by a reading of 7.13, and his doctor recommended a biopsy.
The biopsy showed cancer. Actually, out of 12 biopsy samples only one showed cancer, and that was only in 10% of that sample. The Gleason score was 6. That score is intermediate, between low-grade, well-differentiated tumors that are less likely to progress and high-grade, poorly-differentiated cancers that are more aggressive. With a Gleason score of 6 and a PSA of 7, his cancer would be considered Stage 1, the earliest stage.
Treatment options
For Stage 1 cancer, there are three treatment options: surgery, radiation, and active surveillance. Many of these cancers are so slow growing that the patient will die of something else before the cancer ever causes symptoms or metastasizes. The prevalence of prostate cancer detectable on autopsies rises with age to eventually become almost universal; men are far more likely to die with prostate cancer than of prostate cancer. And the surgery has a high rate of complications, such as incontinence and impotence. The wisest course in many cases is not to treat but to monitor the patient, with the understanding that treatment may be indicated at a later time if there are signs of progression.
Mish’s surgeon strongly recommended surgery, with an estimated price tag of $20,000. His regular doctor recommended radiation therapy, which can be even more expensive and also has a high risk of complications. Mish says he did his own research and found a third option: wait and see. Most doctors would have explained all three options and let the patient participate in the decision. It doesn’t sound like Mish was given the information he needed to give informed consent.
He chose the surveillance option, but he wasn’t content to just sit back and wait. After 60 hours of research, he settled on a cocktail of the ten “most promising things” that he thought would help fight his cancer. His doctor was against the idea but agreed to go along as long as he continued PSA monitoring. He told the oncologist he was going to have a PSA test every month; the oncologist tried to dissuade him, arguing that every six months would be often enough. More frequent monitoring would just detect more meaningless fluctuations. Unfortunately he expressed that by telling Mish the test “wasn’t that reliable,” and Mish misunderstood. He assumed “the more unreliable a test is, the more tests one should take to weed out erroneous outlier results.” But then he got an outlier result and he didn’t weed it out. A scant month later, his score had skyrocketed to 17.65. The oncologist would not have repeated the test that soon, since PSA levels are known to go up after biopsy. A month after that, it had dropped to 2.39, well below the pre-biopsy level and well within the “normal” range. And it stayed down. Eight months after that one high reading, it was down to 1.94. The surgeon wanted another biopsy because of the one high reading. Mish was worried that despite the now low-normal readings, his cancer might have progressed without showing on the test because his cocktail of alternative treatments might have artificially lowered the PSA readings. So he agreed to another biopsy, and this one showed no evidence of cancer.
Mish’s cocktail
Here’s what he took:
     1.  Resveratrol: 250 mg twice daily
     2.  Indole-3-Carbinol: 200mg twice daily
     3.  Bromelain: 500mg twice daily
     4.  Querceten: 300mg twice daily
     5.  Turmeric extract: 300 mg twice daily
     6.  Vitamin D3: 5,000 IU twice daily
     7.  Vitamin K2: 100mcg twice daily
     8.  Vitamin B12: 100mcg twice daily
     9.  Selenium: 50mcg twice daily
     10.  Sodium bicarbonate orally: One half teaspoon twice daily – one hour in the morning right after waking, and again in the evening right before bed – empty stomach
I’ll review these in reverse order:
    • Number 10 clearly contributed nothing; it is foolishness based on Robert O. Young’s nonsensical “pH Miracle” myth.
    • Number 9 may have been counterproductive. There are studies indicating that selenium supplements increase the risk of prostate cancer.
    • Numbers 6-8 are indicated for people with a deficiency of those vitamins, but it is preferable to get them through diet rather than through pills.
    • Number 5, turmeric, has shown promise in preclinical studies, but the concentration needed for its anticancer effects may not be achievable in the human body with supplements.
    • For numbers 2 and 4, indole-3-carbinol and quercetin, a search of PubMed found some rat and lab studies, but no evidence of clinical benefit for human prostate cancer.
    • For number 3, bromelain, PubMed listed no studies about prostate cancer at all.
    • For number 1, resveratrol, there are indeed some studies, but they are far from convincing.
Mish attributes his success mainly to the first four. He particularly has confidence in resveratrol based on some positive studies he found, but he is cherry-picking to find what he wants to believe. It’s easy to find positive studies to support almost anything. If you look for negative studies you can find those too, for instance this one that showed decreased survival with resveratrol. I wonder if his research also dug up the fact that many of the positive resveratrol studies by the most prolific resveratrol researcher were deliberately falsified. He was found guilty of outright fabrication of data, and he was charged with fraud and fired. I consulted PubMed and found two systematic reviews of resveratrol and other phytochemicals. The first concluded:
[it is] impossible to make definite statements or conclusions on the clinical efficacy in cancer patients because of the great variability and differences of the study designs, small patient numbers, short treatment duration and lack of a standardised drug formulation. Although some results from these clinical studies seem encouraging, reliable or long-term data on tumor recurrence, disease progression and survival are unknown. At present, there is no convincing clinical proof or evidence that the cited phytochemicals might be used in an attempt to cure cancer of the prostate.
The other review concluded:
…at present there is no clinical evidence that phytochemicals might have a therapeutic use in prostate cancer in relation to reduction of tumor progression or improved survival. The question about an improved immune function or quality of life remains open. Potentially the use of phytochemicals could play a role in a preventive setting.
And pharmacokinetic studies of doses up to 5 grams per day have demonstratedpoor bioavailability of resveratrol.
In my opinion, Mish’s “research” only demonstrates his incompetence to do research. He went in with an agenda, looking for anything that “might” help and ignoring evidence that it might not. Without a medical or scientific background, he has no way to put the studies he finds into context or to evaluate their credibility, so his judgment is flawed.
What may really have happened
It’s possible to put an entirely different interpretation on the facts Mish provides. Here’s a devil’s advocate version of his story:
There was no need to do PSA screening tests in the first place. The initial elevated readings that led to the biopsy were not signs of cancer; he had a transient rise due to some other, unimportant, cause. The 17.65 reading was an outlier, probably caused by the recent biopsy. The biopsy was not necessary, and the one sample that showed 10% cancer may have only reflected a small focus of disease that would never have caused him any problems. His cocktail of supplements had no effect. In fact, it’s possible that some elements of the cocktail might have been harmful in some way.
Mish (along with a lot of other men) might have been better off if he had never been screened with PSA tests.
Conclusion: Mish probably didn’t cure his cancer
Mish thinks he “beat cancer” with his cocktail of diet supplements; we can’t rule out that possibility, but I think it is highly unlikely. I don’t think his cocktail made a bit of difference. I think he fooled himself. The cocktail fed him false hope, gave him a false sense of control, and provided the virtuous feeling that he was “doing something” rather than passively waiting for what the future would bring.
Fortunately he is still under the care of his regular doctor, who tolerated his experimentation and continued to monitor his condition. The negative biopsy doesn’t mean that his cancer is gone. If it isn’t, we can hope that it won’t progress; and we can hope that if it does, he will get the appropriate treatment.

Note: the original title of this post and the accompanying picture were interpreted by readers as mocking the patient and making fun of him, which was not the author’s intention. The title has been changed and the picture removed in response.

   Clinical equipoise versus scientific rigor in cancer clinical trials
David Gorski
Link
A critical aspect of both evidence-based medicine (EBM) and science-based medicine (SBM) is the randomized clinical trial. Ideally, particularly for conditions with a large subjective component in symptomatology, the trial should be randomized, double-blind, and placebo-controlled. As Kimball Atwood pointed out just last week, in EBM, scientific prior probability tends to be discounted while in SBM it is not, particularly for therapies that are wildly improbable strictly on the basis of basic science, but for both the randomized clinical trial remains, in essence, where the “rubber hits the road,” so to speak. Indeed, when the prior probability of a therapy working based on preclinical basic science investigations appears high, EBM and SBM should be (and are, for the most part) more or less indistinguishable.
The ethics of clinical trials, however, demand a characteristic known as clinical equipoise. Stated briefly, for purposes of clinical trials, clinical equipoise demands that at the time a clinical trial is being carried out there be a state of genuine scientific uncertainty in the medical community over which of the drugs or treatments being tested is more efficacious and safer. One reason (among many) why the Gonzalez trial was completely unethical was a lack of clinical equipose. (Lack of adequate informed consent was another.) Lack of clinical equipoise is also the reason why a prospective randomized, double-blind, placebo-controlled clinical trial testing an unvaccinated group versus a vaccinated control group to determine whether vaccines cause autism would be completely unethical. Such a trial would egregiously violate the principle of clinical equipoise because the unvaccinated group would be left unprotected against potentially life-threatening vaccine-preventable diseases, and that is completely unacceptable from an ethical perspective. Consequently, we have had to rely on on the accumulation of data from less rigorous trial designs to demonstrate that there is no correlation between vaccines and autism. Even so, the accumulated weight of such evidence is enough, and for some questions that is the best we can do because scientific rigor sometimes conflicts with human subjects research ethics. This is an extreme example of lack of clinical equipoise, but it illustrates the point. If we know (or have good scientific reason to suspect) that one treatment is better than another, it is unethical to randomize patients to the arm that receives what is, based on what is known at the time of the trial, likely to be an inferior treatment.
Sometimes, however, the question of whether clinical equipoise exists in a clinical trial is not so obvious as it is for trials proposed by cranks. This situation sometimes crops up in clinical trials for cancer. I was reminded of this issue by a front page story in the New York Times yesterday, New Drugs Stir Debate on Basic Rules of Clinical Trials. In it, reporter Amy Harmon uses a classic human interest story to highlight the issue of clinical equipoise in a clinical trial for a new drug for melanoma that shows great promise. In brief, it is the story of two cousins, one of whom is receiving the new “wonder drug” (whether it is truly a wonder drug or not remains to be seen) in a clinical trial and one of whom is receiving the current standard of care for stage IV melanoma, which, to put it bluntly, sucks in that it has very little effect in prolonging life:
And when, last year, each learned that a lethal skin cancer called melanoma was spreading rapidly through his body, the young men found themselves with the shared chance of benefiting from a recent medical breakthrough.
Only months before, a new drug had shown that it could safely slow the cancer’s progress in certain patients. Both cousins had the type of tumor almost sure to respond to it. And major cancer centers, including the University of California, Los Angeles, were enrolling patients for the last, crucial test that regulators required to consider approving it for sale.
“Dude, you have to get on these superpills,” Thomas McLaughlin, then 24, whose melanoma was diagnosed first, urged his cousin, Brandon Ryan. Mr. McLaughlin’s tumors had stopped growing after two months of taking the pills.
But when Mr. Ryan, 22, was admitted to the trial in May, he was assigned by a computer lottery to what is known as the control arm. Instead of the pills, he was to get infusions of the chemotherapy drug that has been the notoriously ineffective recourse in treating melanoma for 30 years.
Even if it became clear that the chemotherapy could not hold back the tumors advancing into his lungs, liver and, most painfully, his spine, he would not be allowed to switch, lest it muddy the trial’s results.
The melanoma drug in question, PLX4032 (RG7204) was developed by U.S.-based biotechnology firm Plexxikon, which partnered with Swiss pharmaceutical firm Roche to bring the drug to market. The drug targets a specific (but common) mutation in the BRAF proto-oncogene (pronounced “b-raf,” not “braf”) found in melanoma. Interestingly enough, I’ve actually participated in research with a collaborator who is a melanoma surgeon to develop PCR-based methods to detct this very mutation (which was at the time erroneously referred to as the V599E mutation) that results in activation of the BRAF protooncogene, but it’s been five years since I have done any work in this area. In any case, it is gratifying to see that others have developed drugs that target these activating mutations.
Melanoma, for those unfamiliar with the disease, is a cancer that develops in melanocytes, the pigmented cells in the skin. It is highly curable when it is caught early, because surgical excision and sampling of the regional lymph node basins is highly effective. It can even occasionally be controlled in the metastatic setting, resulting in long term survival, if the metastasis can be completely excised surgically. However, once melanoma is stage IV and metastatic to multiple sites or to a site (or sites) from which the tumor cannot be surgically resected, melanoma is incurable. Worse, it’s notoriously resistant to chemotherapy. The very best standard of care chemotherapy regimens include dacarbazine or temozolomide, the former of which is usually first line therapy and the latter of which is sometimes preferred because it can be given orally. Neither work very well.
So in melanoma basically what we have is a disease that is incurable and fatal when it reaches stage IV and for which, unlike the case with, for example, colorectal cancer, there is no good therapy that can markedly prolong survival. Enter PLX4032. PLX4032 is remarkable because it is a targeted agent directed at the V600E BRAF mutation that, in phase I trials, demonstrated significant activity against melanomas harboring that mutation. Since V600E is a common mutation in melanoma, being present in around 50-60% of tumors, that means more than half of patients with metastatic melanoma could potentially benefit from the drug. What do I mean by “significant” activity? In a multicenter dose-escalation phase I trial whose results were recently reported in the New England Journal of Medicine and the press, 81% of patients with V600E mutations showed tumor shrinkage of at least 30%. Moreover, side effects were not prohibitive, with cutaneous (skin) side effects, fatigue, and arthralgias (joint pain) predominating. The worst potential side effect was squamous cell carcinoma of the skin.
These results are quite impressive for a phase I study. Remember that phase I trials are not designed to detect efficacy; they are safety studies. The idea is to increase the dose in humans until dose-limiting toxicities are encountered. If tumor responses are seen, so much the better. Typically, less than 25% of subjects in a phase I trial will demonstrate measurable tumor shrinkage, making 81% very impressive indeed. Moreover, among subjects receiving the drug, the estimated median progression-free survival (PFS) has been 7 months, compared to historical controls of around 2 months. The duration of these responses was on the order of 8 months, which means that the tumors did nearly all start growing again, but, even so, this is far better than the standard of care chemotherapy for metastatic melanoma, dacarbazine, which only slowed tumor growth in 15 percent of patients for an average of two months. Of course, the results of this phase I trial are no guarantee that the apparent increase in PFS (which may or may not be real, given that there is no control arm in a phase I trial) will translate into an increase in overall survival (OS) — remember Avastin — but it is very promising, which is why this new clinical trial was begun. Indeed, I view PLX4032 as a prime example of SBM at work, starting with a basic science observation (many melanomas harbor BRAF mutations that drive their growth) and using that basic science to develop a therapy, ultimately bringing that therapy to clinic.
So here’s the question. Given the result of this phase I trial, is there truly clinical equipoise regarding PLX4032 in melanoma that has the V600E BRAF mutation? It’s not an easy question, and some physicians come down on either side of the issue, as discussed in the NYT article. First, the side arguing that drugs like PLX4032 are challenging our current cancer clinical trial paradigm:
But critics of the trials argue that the new science behind the drugs has eclipsed the old rules — and ethics — of testing them. They say that in some cases, drugs under development, PLX4032 among them, may be so much more effective than their predecessors that putting half the potential beneficiaries into a control group, and delaying access to the drug to thousands of other patients, causes needless suffering.
“With chemotherapy, you’re subjecting patients to a toxic treatment, and the response rates are much lower, so it’s important to answer ‘Are you really helping the patient?’ ” said Dr. Charles L. Sawyers, chairman of human oncology at Sloan-Kettering. “But with these drugs that have minimal side effects and dramatic response rates, where we understand the biology, I wonder, why do we have to be so rigorous? This could be one of those defining cases that says, ‘Look, our system has to change.’”
But does it? On the other side:
Defenders of controlled trials say they are crucial in determining whether a drug really does extend life more than competing treatments. Without the hard proof the trials can provide, doctors are left to prescribe unsubstantiated hope — and an overstretched health care system is left to pay for it. In melanoma, in particular, no drug that looked promising in early trials had ever turned out to prolong lives.
PLX4032 shrinks tumors in the right patients, for a limited time. But would those who took it live longer? No one knew for sure.
“I think we have to prove it,” said Dr. Paul B. Chapman, a medical oncologist at Memorial Sloan-Kettering Cancer Center who is leading the trial. “I think we have to show that we’re actually helping people in the long run.”
Both are powerful arguments. On the one hand, if drugs like PLX4032 really are far and away more effective than previous generations of experimental drugs in the pipeline to market, then rigidly sticking to the old system has the potential to result in the loss of potentially savable lives and in increased suffering that could potentially have been ameliorated. However, we have been fooled before. Drugs that look highly promising in preliminary studies have ultimately fizzled, and increases in response rates and even increases in PFS have not always translated into increases in OS. Again, remember Avastin. Then, as was suitably mentioned in the NYT article, there is the cautionary example of bone marrow transplantation for advanced breast cancer, which during the 1990s was thought to prolong survival. As a result, both physicians and patient advocacy groups lobbied, using their political muscle to persuade health insurance companies to pay for bone marrow transplants for breast cancer. However, when the careful clinical trials were done, it was found that bone marrow transplantation was no more efficacious than high dose chemotherapy and that it caused death in some cases. The history of cancer research is littered with drugs that appeared efficacious in early clinical trials and then failed when subjected to more rigorous testing. How do we know that PLX4032 isn’t one of those drugs?
We don’t. Clinical trials are how we figure out the answer to that question:
“My goal,” Dr. Chapman shot back, “is to find out as quickly as possible in as few patients as possible whether this works. If we never know, then we’re never going to be able to build on anything.”
One of the melanoma field’s senior clinicians, Dr. Chapman had lived through trial after trial of drugs that failed to live up to early promise. Almost every oncologist knew, too, of a case nearly 20 years earlier when bone marrow transplants appeared so effective that breast cancer patients demanded their immediate approval, only to learn through a controlled trial that the transplants were less effective than chemotherapy and in some cases caused death.
“Making patients’ tumors go away is gratifying,” Dr. Chapman told critics. “But that’s not the business I’m in. I’m in the business of making people live longer. That’s what I want to do.”
Which brings us back to the whole debate on whether PFS in the absence of OS is reason enough to approve a new anticancer drug, the very same question faced for Avastin. Again, we have no idea that PLX4032 even prolongs OS compared to the standard of care, although we do have an indication that it does prolong PFS. On the other hand, PLX4032 has considerably less toxicity than dacarbazine, meaning that the potential to cause harm would seem to be less, at least from what we know now. All of this leads to a critical question: Is a phase I trial with stellar results enough to claim that there is not equipoise between PLX4032 and the standard of care? Add to that information in the NYT article that describes the “Lazarus” effect observed in some patients who were clearly weeks or even days from death could get off of oxygen and out of bed after placed on PLX4032, sometimes even for months. Is this all enough information to destroy clinical equipoise for this drug and make performing a randomized trial against standard of care unethical?
One thing that puzzled me initially is why the design for this trial was chosen. It’s a straightforward open label randomized trial comparing PLX4032 against dacarbazine that does not allow patients in the dacarbazine arm to cross over if they are receiving no benefit. Originally, the principal investigator of the current trial wanted to do a test of PLX4032 versus standard of care in only the sickest patients. If, reasoned Dr. Paul Chapman, the drug did indeed show this “Lazarus effect” in some patients, it would be justification to get it approved as rapidly as possible, even before evidence of improvements in OS were shown. On the other hand, it appears that the drug company (Roche) feared that such a trial would only provide justification for approval for PLX4032 only in that small group of the sickest patients. It wanted approval for the widest indications possible, which requires a large, phase III randomized clinical trial.
In other words, economics appears to have trumped science and ethics. The best design from an ethical standpoint, one that would also be scientifically strong, would have been to do a randomized trial of dacarbazine plus or minus PLX4032. One group would receive dacarbazine plus placebo; the other dacarbazine plus PLX4032. Both groups would be receiving the standard of care, and there would still be genuine uncertainty whether the combination is better than dacarbazine alone. This is how clinical trials testing many new anticancer drugs are performed these days. Because clinical equipoise would tell us that testing a new anticancer drug versus placebo would be unethical in most cases, the two choices left are to test a new drug either against standard of care or added to standard of care. In addition, for trials in which clinical equipoise is not as clear as it should be, in general there is a built-in planned interim analysis of PFS and OS, wherein the trial is stopped if one group is doing significantly more poorly than the other group to the point where the results are so clear-cut that statistically there is no way that the addition of the remaining patients could change the results. In addition, often patients are allowed to cross over to the other group if after a certain period of time they are receiving no benefits. This can go both ways, either control crossing over to experimental drug or those receiving experimental drug crossing over to the control group. Such a design makes the analysis of the data harder to do, but it does mitigate some of the ethical concerns. Unfortunately, in this case, the investigators chose clarity rather than allowing patients to cross over.
Interestingly enough, it is the basic scientists who are most in favor of the contention that this drug should be more widely available:
Some of the strongest criticism came from laboratory researchers who study the biology of the disease and see the drug as fundamentally different from its predecessors. The previous red herrings, they argued, never had such a high response rate. Few other drugs had shrunk tumors in as high a percentage of patients with melanoma or any other solid tumor as PLX4032 had in its first human trial.
“Many of my colleagues who are outstanding clinical investigators have been able to convince themselves that this is a fair thing to do,” Dr. David E. Fisher, a leading melanoma biologist at Massachusetts General, said of the controlled trial. “My personal view is it’s nuts. I don’t know anyone who hasn’t shuddered at the concept that we can’t let patients on the control arm cross over because we need them to die earlier to prove this point.”
Quite frankly, even though I am a staunch advocate of science-based medicine, I, too, shudder at this design. On the other hand, the basic scientists seem rather cavalier to me in their belief that improvements in response and PFS will translate into improvements in OS. After all, even though the toxicities observed in the phase I trial were relatively mild, one of them was squamous cell carcinoma. As promising as PLX4032 is, there is still power to the argument that we can’t know for sure if it is superior unless we actually do the careful clinical trial, keeping the failures of promising treatments of the past in mind.
In oncology clinical trials, as in clinical trials for treatments of any life-threatening disease, there is always a tension between wanting the “cleanest” possible results versus doing the best for each individual patient. It is a balancing act that relies on the ethics of physicians and a combination of hope and altruism in the patients who become subjects in such trials. Both patients and physicians want the drugs being tested to work, but in some cases clinical trials are very much, to invoke my Star Trek geek tendencies, a case where the good of the many is weighed against the good of the few — or the one — with the balancing of these factors done using knowledge with huge gaps in it. How to maximize the good for as many patients as possible is the goal, but, as we have seen, this is a goal that is not so easily accomplished, just as clinical equipoise is a concept that is easily stated but not so easily applied. PLX4032 teaches us that.

   When impressive science fails to impress patients
David Gorski
Link
One of the greatest challenges in medicine can sometimes be to convince patients that the results of scientific and medical research apply to them, or, at the very least, to explain how such results apply. One of the reasons that medicine based not on science or evidence fluorishes is because it can be so hard to explain to patients why a particular intervention is viewed as effective. My co-blogger Steve Novella wrote about some of the fallibilities of human perception that lead to perceiving correlations and treatment effectiveness where there are none. R. Robert Bausell wrote about the same thing in his recent book Snake Oil Science. While it is undoubtedly true that people tend to pay more attention to anecdotes than to studies and statistics, there is also another reason why doctors often have problems convincing patients of the value of health interventions, and that’s the difference in perception and how we value different kinds of evidence.
A couple of years ago, I came across an article that explains this gulf between how those of us trying to practice science- and evidence-based medicine perceive the world and how most human beings not trained in medicine or science perceive it. The article, which was published in 2006 in the New York Times and written by Dr. Abigail Zuker, proposed one reason why this might be, beginning with a discussion with her mother in which she tries to convince her of the benefit of exercise, even in the elderly, a concept that her mother would have none of and dismissed contemptuously:
“Studies,” she says, dripping scorn. “Don’t give me studies. Look at Tee. Look at all the exercise she did. She never stopped exercising. Look what happened to her.”
End of discussion. Tee, her old friend and contemporary, took physical fitness seriously, and wound up bedbound in a nursing home, felled by osteoporosis and strokes, while my mother, who has not broken a sweat in the last 60 years, still totters around on ever-thinning pins. So much for exercise. So much for studies. So much for modern clinical medicine, based on the randomized allocation of treatment and placebo. All that beautiful science, stymied by the single, incontrovertible, inescapable image of Tee, the one who exercised but grew hunched and crippled anyway.
My first thought would be that such a reaction represents the power of anecdote over clinical data, but Dr. Zuker sees more than that. She sees it as the difference between how doctors are trained to view the world and how people untrained in medicine and science view the world, and she uses a rather interesting metaphor to convey this difference:
It is medicine’s eternal quest, these days, to sell impressive science to unimpressed patients, and it is hard to think of a group less equipped to do it than doctors. Doctors are specifically trained not to think like normal people, not to see what others see or to reason as others reason. They — er, we — come to operate in an atmosphere so thin, so heady and attenuated with the power of statistical analysis, that one might wonder whether we are really on the same planet as the patients we try to convince of our truths.
“Exercise helps the elderly.” The doctor sees, from a perch suspended somewhere up in the sky, a large football field filled with the elderly. There are thousands of them down there, all dressed in sweats and sneakers, dumbbells at their feet. Half of them are using the dumbbells, or are down on their backs, doing leg lifts. The others just stand around.
Over the years, of course, the ranks thin. The doctor watches, counts. It begins to look as if there are more exercisers left. After decades, there are definitely more exercisers. Of course, there are still a few sloths standing around (and one of them looks suspiciously like my mother). But by and large, the exercisers come to rule the field.
That is the view from on high. Down on the field, of course, the view is quite different. You are standing in a thick crowd, minding your own business, living your life, but you cannot help noting that the man over there threw his back out with all that exercise, and the woman next to you, grunting to lift her dumbbell, had a heart attack. You cannot see to the other end of the field and have no idea what is happening there. But watching all the sweating and grunting and seeing some of those exercisers disappear anyway, you decide to opt out.
This is one of the best metaphors for the gulf between how we as practitioners of science-based medicine and patients see clinical and epidemiological research that I’ve ever come across. The aspect of clinical research that is hard to explain to patients is that interpreting the results of clinical studies and applying them to patients is entirely a matter of probabilities. From clinical studies, you can conclude that exercise will increase a person’s chance of living to a ripe old age. However, some people, either through good genes or sheer luck, manage to make their way through life without exercising and live to a ripe old age anyway. (Think George Burns, who lived to 100 despite a love of cigars and booze.) Similarly, we can say that, by and large, smoking can greatly increase your risk of heart disease, lung cancer, and premature death. But, then, everyone knows someone who smoked for 70 years and died at age 85 of old age, apparently suffering few ill effects from the thousands upon thousands of cigarettes he smoked. Indeed, heavy smokers have “only” approximately a 25% lifetime risk of developing lung cancer. That leaves lots of smokers out there who never develop lung cancer and makes it easy for some to ignore the fact that heavy smoking increases one’s risk of developing lung cancer by a huge amount and that lung cancer was considered a rare disease before the advent of cheap mass-produced cigarettes. Now it’s consistently in the top two or three causes of cancer death, and that appears to be entirely due to tobacco use.
It’s sometimes very hard to overcome the power of anecdote and patient experience, even for such clear-cut cases. For example, take the rather common discussion of the pros and cons of breast conserving therapy (lumpectomy) versus mastectomy for breast cancer. Every surgeon who does breast surgery will get the occasional patient who demands a mastectomy for a small tumor that would be most appropriately treated with lumpectomy. In my experience, the reason almost always boils down to a bad personal experience. Often such patients had a close relative or friend who had breast cancer treated with lumpectomy and recurred. Often, if you probe a little more closely, they will tell you the horrific tale of how the cancer ravaged a loved one after it recurred. They become convinced that it was because the deceased hadn’t undergone the more radical treatment of mastectomy that her cancer killed her. No amount of citing the three decades of large studies demonstrating that five and ten year survival are the same for lumpectomy and radiation therapy as they are for mastectomy will sway them. The patient doesn’t care; she wants a mastectomy. This is even more true when you inform her that there is around an 8% chance of local recurrence with lumpectomy but that it doesn’t affect the overall survival rate. All she hears is that there is nearly a one in ten chance of the cancer coming back in her breast after a lumpectomy but a less than 1% chance of its doing so after a mastectomy. Statistics tell the doctor that her chances of long-term survival will be the same with a lumpectomy as with a mastectomy. The patient’s experience tells her otherwise. Therefore, she insists on a mastectomy, and, because it is a perfectly acceptable treatment for breast cancer and because she has the right to control what happens to her body, she usually gets it, even though usually more than one surgeon try to talk her out of it and even though she could surely have been treated for her cancer without losing her breast. This same phenomenon seems to be progressing, at least from my–forgive me–anecdotal experience. Women with small cancers seem to be requesting more and more to have both of their breasts removed, even though such a prophylactic measure can be justified scientifically primarily in women with a genetic mutation or a family history that puts them at a very high risk of breast cancer and such operations are not without a cost in complications and decreased quality of life. Such a trend away from less radical surgery towards even more radical surgery is also being fed by us physicians as well in the form of ever more sensitive imaging modalities like MRI that find ever smaller cancers that may never develop into life-threatening disease or find small pockets of disease that normally wouldn’t have been detected before and would have been “mopped up” with post-lumpectomy radiation therapy.
It is this gulf in perception that will lead some patients to refuse chemotherapy even when there is clear-cut evidence of a significant survival benefit or sometimes even in the case when it is the only curative treatment available. When this occurs, it is often because the patient either knew someone who underwent chemotherapy and died anyway, even if the cancer was a high mortality cancer that chemotherapy had a small chance of curing, or someone who had a complication from chemotherapy, such as neutropenic enterocolitis, and died from the complication rather than the tumor. From their “ground level” perspective, they don’t see that the number of surviving patients with this particular cancer at this particular stage who receive chemotherapy is far larger than the number of those who don’t receive chemotherapy. They may not even realize that in the case of some tumors (leukemias and lymphomsa, for instance), virtually no one who is not treated survives his or her cancer. All they see is that someone they knew had chemotherapy and died from it. Indeed, various “alternative medicine” sites do their best to feed this perception by emphasizing the complications of chemotherapy, using terms such as “poisoning” and “burning” and making incorrect claims that chemotherapy harms more patients than it helps.
If the power of anecdote can be difficult to overcome in such clear-cut cases, imagine how hard it can be to do in cases where the decisions are not so clear cut or where the treatment proposed offers only a small benefit. One such example is whether or not to use adjuvant chemotherapy after the treatment of various cancers, where, depending upon the cancer and the stage, the increase in absolute chances of survival can range from as little as 3% to as much 20-25%, but rarely more. In the case of stage I breast cancer, for example, the absolute survival benefit due to adjuvant chemotherapy is on the order of 3%. Even so, most patients will still opt for even this modest benefit. Consider next the case where the literature and studies are either lacking, of poor quality, or multiple and in conflict, and you can see why patients, not to mention physicians, will base their decision more on personal experience than science. All of these are barriers to practicing science-based medicine.
Dr. Zuker suggests some strategies to persuade the patient:
Good doctors learn some tricks, over the years, to let patients see what they see. It helps, sometimes, to descend part of the way down from the sky and give a smaller version of the big picture. (“Of all my patients, it’s the ones who exercise who do the best.”) Sometimes it helps to get down completely, and see what the patient sees (“Your grandmother smoked till she was 90, but you may not be so lucky.”)
But sometimes there is no convergence of views. The patient who sees only from the ground, the doctor who sees only from the sky may simply have to agree to disagree, and have the same dialogue over and over again.
Another way we in medicine can overcome this gap, it is suggested, is to develop tests that help us predict far better than we can which patients will respond to which drugs and which patients are at higher risk for various complications. This “personalized medicine,” based on genomics and other factors, is at present a hot area of research, but I’m not entirely convinced that it will narrow gulf between patient and physician perceptions that much, because even if these tests can produce more precise information upon which to base treatment choices, it will still come down to a matter of probabilities. It will still come down to a matter of the gulf between the perception of physicians looking at the stadium from the air and the patients at ground level. The only difference will be that we’ll be saying “patients with this genomic profile” are more likely to respond to a treatment than those without. I’m not sure that will help bridge the gap that much.
Of course, I would point out to Dr. Zuker that it is not only patients who can fail to see the forest for the trees. Doctors often fall into this trap as well. If, for example, we see a treatment resulting in a bad outcome in two patients in a row, we are just as prone as anyone to react by changing our approach, even when the scientific literature tells us that we are using the most efficacious treatment available and that the last couple of bad outcomes were almost certainly a statistical fluke. At the clinic level, dealing with real patients and real complications, we physicians often can’t see the “whole picture” either, and that is one reason why it is so important for us to stay current on the medical literature and to be willing to reach out to their colleagues for advice. In my experience, surgeons tend to be particularly prone to this “last disaster” sort of thinking. Sometimes this is a good thing, as in when a string of wound infections leads one to wonder if one’s sterile technique is not what it should be, but it can also lead to major changes in treatment choice based on anecdotes.
In the end, bridging this gap in perception is largely a matter of communication. This is one subject for which doctors are generally not well trained, either in medical school or residency, and one area that I wish I had become better at. In a sense, even now, I am still learning and still trying to improve. However, as Dr. Zuker points out, often that isn’t enough, and, as I like to point out, it is this gulf that makes it all too easy for practitioners of dubious unscientific medicine to persuade patients that their unproven or ineffective nostrums are superior to science-based medicine.

   Persistence of Memory
Mark Crislip
Link
I have steadily endeavored to keep my mind free so as to give up any hypothesis, however much beloved (and I cannot resist forming one on every subject), as soon as the facts are shown to be opposed to it.
— Charles R. Darwin
I’m getting old: 50, almost 51, and that’s over 350 in dog years. As a result of my advancing age there are things I do not understand: tattoo’s, hip hop, visible undergarments, and those rectangular, square plastic glasses that seem so popular and look hideous on everyone. It gets harder to change.
I have been able to stick MD after my name for almost a quarter century now (175 dog years for those keeping track), and it does give a sense of perspective to the ebb and flow of medical therapies. Medicine for the last hundred years has been all about change. Dogma from last century is nonsense this century, all due to that damn science. It gets so tiresome having to learn something new.
Last month’s New England Journal of Medicine was another in a seemingly endless series of plus ça change, plus c’est la même chose moments.4 They published the results of the CORTICUS study, a trial that looked at the use of corticosteroids in septic shock. For background, septic shock occurs when a severe infection shuts down most important organs and the patient cannot maintain their blood pressure. Mortality is high and, depending on the patient, the infection and the infecting organism, mortality ranges from 30 to 110%. People can die several times during their admission and be temporarily brought back, at least from the sloppy definitions of the near death experience proponents (shameless plug for Quackcast 23). Sepsis is a Bad Disease. Patients in septic shock often have lower levels of cortisol in their blood than they should. So maybe giving replacement cortisol, in the form of hydrocortisone or prednisone, would decrease the death rates.
The results of the CORTICUS (I so want to be involved in a study that has an obscene acronym) suggests steroids are not effective. When compared to placebo, physiologic hydrocortisone (i.e. giving enough hydrocortisone to replace the deficit) was no better at preventing mortality. It didn’t work. Or didn’t it. Or did it? Which one?
I remember back when I was a resident back last century. We had to walk to the hospital barefoot in the snow, uphill, even in August. Made our own penicillin in the basement from moldy cheese. We took call every day and loved it. The sun was warmer, the sky bluer, water was wetter. Good times. It was also when the first study of sepsis and steroids was published. In that study they used relatively large doses and had an improved outcome.5
So everyone got steroids for sepsis.
Then another, better study, was published that demonstrated no change in mortality, but the patients who received steroids had more secondary infections. Bummer. More data. I have to change my practice. Damn. I hate it when that happens.
So the use of steroids faded. In medicine every therapy returns in a new way. I am waiting for the resurgence in the use of theophylline, for example. It’s going to happen.
Then it was noted that septic patients were relatively cortisol deficient. When tested their cortisol levels were lower than it should be in sepsis and that their adrenal function, the source of cortisol, was suppressed. Perhaps if patients received physiologic, replacement doses of steroids, they would do better.
Early studies looked promising, several small trials were done and several meta analyses suggested that there was a sweet spot for steroids, that indeed a lot was dangerous but a little was of benefit in decreasing mortality.
Although meta analyses are often subsequently disproved, it was compelling information. So steroids came back in vogue. Damn. Have to change practice again. Given that my mind is mostly filled with the lyrics of 1970’s rock music, it better not happen again. There is just so much I can learn.
Until the CORTICUS study. Now what. Do we give steroids? Or not? Or in a subgroup of patients? Damned if I know. My Critical Care colleagues tell me that they may yet be a role in patients with refractory sepsis since cortisone made them better faster, and generally the quicker you get out of a septic state the better you do (although not demonstrated in the CORTICUS study, it was not set up to determine outcome in this subgroup so the issue is still open to debate).
So over the last 25 years its yes and no and yes and mostly no when it comes to the treatment of sepsis.
This ebb and flow of medical knowledge is a standard feature of the practice of medicine and is, to the best of my knowledge, never a feature of sectarian or unscientific medicine (so-called Complementary and Alternative or Integrative Medicine).
I often wish I could practice unscientific medicine. Learn one all encompassing myth and never have to change again. CAM proponents often point to the changing interventions of medicine as a flaw. Look, they say, these guys don’t know what they are doing. One day its yes, next day its no, then its yes again. Do you know why a doctor’s mind is so clean and pure? Because she is always changing it.
Part of the problem is that in medicine we learn and practice evolves as the science behind the practice changes, and as an ever moving target, we are always refining practice. It is a strength of medicine, but leads to a degree of doubt and uncertainty and one thing people hate is doubt and uncertainty. Better a firm conviction about nonsense than a doubt about reality.
The other issue this study raises, and it is an important one when reading in clinical study, is sample size and its influence on the ability to determine differences in study groups. The smaller the study size, the more difficult it is to find a difference and the more likely it is that differences are due to noise. This concept is expressed as the power of a study.
The power of a study can be calculated in advance. In the accompanying editorial they note, given a 35% mortality rate, that to demonstrate a 15% decrease in relative risk of mortality would require 2600 patients. The CORTICUS study had 499 patients and would not be able to accurately determine small differences between the groups.
Note that to find a 15% difference, you need to study 2600 patients.
This is an issue in most clinical studies in both science-based medicine and in studies of sectarian modalities: the sample size is so small that “statistically significant” results are more likely due to background variation. If you have a small difference that is statistically significant between small groups, it is probably meaningless.
Look at the recent acupuncture and heartburn study:2 15 patients in each group. Even if the difference in the two groups were 100% and the end point was death i.e. 15 lived and 15 died, the results would still be meaningless, regardless of the statistical significance. Not enough patients.
Most studies, especially in the CAM literature, suffer from this issue. Most are simply extended case series. It’s the old joke.
One patient with a disease: “In my experience…”
Two patients with a disease: “In many patients…”
Three cases with a disease: “In case after case after case…”
BTW. Read Prior Probability: The Dirty Little Secret of “Evidence-Based Alternative Medicine” on this blog. Makes me feel like M.J. GUMBY: “Oh I have a piece of brain stuck in my head… oh my head hurts!” It’s what all statistics do to me. But it is a key concept in reading any medical literature, CAM or not.
Unfortunately, the problem with information, good, bad, or indifferent, is once it gets in the literature it stays. There are a lot of myths in the practice of medicine: normal temperature is 98.6, atelectasis causes fever, you have to double cover Pseudomonas, antibiotics are strong or powerful. The problem can be that with any strongly held opinion, it is hard to change your mind. Human psychology is such that people look for facts that confirm pre existing beliefs and ignore those that contradict them. Medicine is no different and it is interesting how misinformation persists when it is contradicted by good data.
The Journal of the American Medical Association published an article last month entitled “Persistence of Contradicted Claims in the Literature.”3
They looked at data that was initially based on observational epidemiology that were later contradicted by randomized clinical trials. In this case they looked at Vitamin E and cardiovascular benefits and Beta Carotene and Alzheimer’s.
Epidemiological studies suggested benefit from each, later clinical trials said, nope, sorry, we were mistaken.
It seems that despite excellent clinical trials, people cannot bring themselves to refute the older epidemiologic data and persist on citing the literature in support of these interventions that don’t work.
What is also interesting is the rationales for not believing the controlled trials, ranging from quibbles bout bias, differing populations, pharmacokinetics, confounding variables to poor endpoints.
So I suppose steroids will continue to be used in sepsis, depending on the patient and how the doc understands a complex and changing literature.
This article also brings some insight into all medicine, real or “alternative.” If you are convinced, for whatever reason, that you can treat a disease with water or needles or even, heaven forbid, steroids, it may be difficult to change your mind even if the contradicting data is good. It is how people are.
This brings home the issue as to how difficult it is to do convincing clinical trials: even the best trials can be unconvincing if you are unable to discount earlier, poorly designed or less powerful, studies. Bad trials, like those found in sectarian practices, crumble under the slightest touch. Once you read past the self aggrandizing abstract (most do not) you see that, like Oakland, there is no there, there.
In sectarian medicine, early trials, which tend to have positive results, are also the more poorly designed, biased trials. Once published, they persist in supporting unscientific modalities even after better trials demonstrate the lack of efficacy. The arc of publishing studies related to sectarian modalities is increasingly well done studies with decreasing efficacy until an excellent study shows no effect. But the early studies continue to pollute the intellectual waters. I wonder if there is such a thing as intellectual primacy: the first thing you learn, regardless of subsequent contradictory information, sticks better in the brain solely because it was the first learned.
It must be nice to be a CAM provider, or some MD’s I know: immune from ever having to learn or change.
Take home?
Medicine is hard, confusing, contradictory and always changing. Real docs may share more commonalities with CAM practitioners than they would like. Change is hard, especially for old fogey’s like me.
Makes we wish I were a homeopath.
References
Sprung CL, Annane D, Keh D, Moreno R, Singer M, Freivogel K, Weiss YG, Benbenishty J, Kalenka A, Forst H, Laterre PF, Reinhart K, Cuthbertson BH, Payen D, Briegel J; CORTICUS Study Group. Hydrocortisone therapy for patients with septic shock. N Engl J Med. 2008 Jan 10;358(2):111-24. PMID: 18184957 Return to text
Aliment Pharmacol Ther. 2007 Nov 15;26(10):1333-44. Epub 2007 Sep 17. Clinical trial: acupuncture vs. doubling the proton pump inhibitor dose in refractory heartburn.
Dickman R, Schiff E, Holland A, Wright C, Sarela SR, Han B, Fass R. Return to text
JAMA. 2007 Dec 5;298(21):2517-26. Persistence of contradicted claims in the literature. Tatsioni A, Bonitsis NG, Ioannidis JP. Return to text
The more things change, the more things stay the same. Barbarian. Return to text
A meta analysis: http://www.annals.org/cgi/content/full/141/1/47. A structured review: http://www.bmj.com/cgi/content/full/bmj;329/7464/480

   Compare and Contrast
Mark Crislip
Link
I have been in Infectious Diseases for almost 25 years. I have two major jobs: I see inpatient consults and I chair the Infection Control program. I have been involved in quality improvement, especially as it relates to hospital acquired infections, for my entire career. It has been an interesting quarter century. Year after year we have driven down infection rates and other kinds of mortality and morbidity in hospitalized patients. Everyone recognizes that medicine is difficult and dangerous and its biggest problem is medicine is practiced by humans, who, I would venture to observe, are prone to mistakes and any number of cognitive errors.
It has not been a easy journey. People hate change and there has not always been certainty as to the best options to choose to solve a problem, a problem that continues today. For example, how best to treat a patient with potential methicillin resistant Staphylococcus aureus colonization (MRSA). Should we screen everyone? Screen high risk patients? Surgical patients? Do we decolonize, with the long term consequence of accelerating antibiotic resistance? Do we place everyone with MRSA in isolation, with the known decrease in care that patients in isolation may have? Everything we do has potential downsides and unintended consequences. No good deed ever goes unpunished.
When I was a resident every PVC (preventricular contractions) in cardiac patents was suppressed as we thought PVC’s were the sentinel event that led to ventricular tachycardia and death. So patients received IV lidocaine and we often sent patients home on quinidine or other antiarrhythmics. Subsequent studies demonstrated that antiarrhythmics may have killed more people than they saved, and doctors no longer suppress every PVC in the ICU. Medicine changes, one hopes for the better, offering old geezers like me the opportunity to ‘reminisce’ about the old days, when I tied an onion to my belt, which was the style at the time. Now, to take the ferry cost a nickel, and in those days, nickels had pictures of bumblebees on ‘em. Give me five bees for a quarter, you’d say. Now where were we? Oh yeah: the important thing was I had an onion on my belt, which was the style at the time.
Sorry. I digress. A couple of months ago I had a patient with severe malaria that needed IV quinidine (the dextrorotatory diastereoisomer of quinine, but that is obvious) and there was none in the pharmacy; we had to get some shipped in by our dealer, er, distributor. In medicine, if it discovered that what was thought to be a beneficial intervention turns out to cause more harm than good, the intervention is abandoned.
When To Err is Human was published, it caused quite a brouhaha. Depending on whose numbers you want to believe, 44 to 98 thousand Americans died each year from medical errors. I did not have issues with the numbers. There are about 5700 hospitals in the US, so that would be about one death every month and a half. Knowing what I did about infections and other complications in the hospital, that did not seem like an unreasonable estimate. A bit high, perhaps, but in the ballpark of my understanding of reality.
That is old data, and no longer applicable. Due, in part, to To Err is Human, the last decade has led to innumerable studies evaluating the causes of infections and complications in hospitals and the best approaches to decrease them. My institutions have invested huge time and effort to implement these quality improvements with great success.
When I started in Infection Control, the infection data was considered protected. The thought was that institutions would be more likely to collect and evaluate data about infections if there were not discoverable by lawyers. The downside was, as I was informed, if I informed anyone about the data, it would no longer be protected, and the institutions would be open to expensive lawsuits. This century my institutions are much more transparent about disseminating information about our practice. I thought I would have a conniption the first time I saw the infection data for the ICU posted in the ICU for all too see. It turns out, like so many deeply held convictions, that keeping the data protected was a bad idea. Transparency has not led to an increase in lawsuits but it has led to a decline in all manner of hospital associated complications. Our staff takes a great deal of pride in their work. They took the data as a personal affront and worked to improve all aspects of patient care. When they saw harm potentially occurring, practice changed for the better.
So there are three reactions to new data in medicine that demonstrates that a given medical practice may cause harm.
First, the data and the conclusions are challenged, as they should be. All studies are open to analysis and improvement. In medicine we continuously try to improve care, and that requires good information.
Second, further studies are done to confirm and refine the problem and other studies are done to see how practice can be improved.
The third is practice change, which is often slower than we like. But change we do. I am old enough that I often bore the residents with how it used to be in the old days. Medicine today is drastically different than 20 years ago, and many logs better.
The most impressive example of improvement in the last 20 years is hand hygiene. Hand washing adherence 20 years ago was an embarrassment. Part of the problem was the time it took to wash hands; I have heard that if a nurse spent her time appropriately washing her hands, 80% of his shift would be spent at the sink.
Alcohol foam changed that. When we introduced the foam in the hospital, even when compliance was only 20%, we had a 50% drop in hospital infections, and as compliance has increased to 90 to 95%, the infections had a parallel decline. And the effects of alcohol foam improved once I discovered it was not a po agent, to be used like Cheez Whiz. Our biggest problem now is fall prevention. Most falls occur when the patient doesn’t want to bother the nurse and, in the process of moving about the room without help, falls. We are investigating ways to remove gravity from the hospital.
The threshold for changing and abandoning a therapy can be very small. Last century there was a drug called trovafloxacin that caused several hundred cases of hepatitis and a half a dozen deaths when it was being prescribed at a rate of 300,000 new prescriptions a month. Hepatitis was an extremely unusual complication, but the small risk was not worth the potential benefits since there were equally efficacious alternatives.
Compare and contrast medicine and alt med.
Most alt med interventions are, of course, based on eternal truths that cannot be improved or changed. They are often immune to reality induced change. Studies that confirm their eternal truth are always accepted. Studies that show harm or lack of efficacy? Not so much.
Certainly, when complications of alternative medicines are published the data is up for discussion. That is good and as it should be. But that is where the similarities with medicine ends.
There is an ongoing issue of safety in the two most invasive alternative interventions: acupuncture and chiropractic.
There were two reviews concerning chiropractic safety published recently. Safety of chiropractic interventions: a systematic review, which found
A total of 376 potential relevant articles were identified, 330 of which were discarded after abstract or complete article analysis. The search identified 46 articles that included data concerning adverse events: 1 randomized controlled trial, 2 case-control studies, 7 prospective studies, 12 surveys, 3 retrospective studies, and 115 case reports. Most of the adverse events reported were benign and transitory, however, there are reports of complications that were life threatening, such as arterial dissection, myelopathy, vertebral disc extrusion, and epidural hematoma. The frequency of adverse events varied between 33% and 60.9%, and the frequency of serious adverse events varied between 5 strokes/100,000 manipulations to 1.46 serious adverse events/10,000,000 manipulations and 2.68 deaths/10,000,000 manipulations.
CONCLUSION: There is no robust data concerning the incidence or prevalence of adverse reactions after chiropractic. Further investigations are urgently needed to assess definite conclusions regarding this issue.
That is impressive complication rates, although the authors suggest the data to support the rates are not robust, for an intervention that only has at best proven efficacy for low back pain and safer alternatives. Also published recently was Deaths after chiropractic: a review of published cases.
Twenty six fatalities were published in the medical literature and many more might have remained unpublished. The alleged pathology usually was a vascular accident involving the dissection of a vertebral artery.
That is about three times the number of deaths from trovafloxacin, an excellent antibiotic that we abandoned in the U.S. as too dangerous. Of course, we have safer alternatives with equal efficacy.
Also recently published was Acupuncture-related adverse events: a systematic review of the Chinese literature which found
that in total reported on 479 cases of adverse events after acupuncture. Fourteen patients died. Acupuncture-related adverse events were classified into three categories: traumatic, infectious and “other”. The most frequent adverse events were pneumothorax, fainting, subarachnoid haemorrhage and infection, while the most serious ones were cardiovascular injuries, subarachnoid haemorrhage, pneumothorax and recurrent cerebral haemorrhage.
Based on the reported complications of the two interventions, if they were a medical therapy regulated in a manner similar to medications and medical devices, they would certainly have, at a minimum, a black box warning and, in the case of chiropractic, no longer be used. Especially as there are no good indications for chiropractic or acupuncture.
What you do not see in the medical literature or the chiropractic blogs is any concern that harm may be done and investigations into changes in practice that could minimize the morbidity and mortality.
Instead you get The Self Importance of Being Ernst and Death by Chiropractic Another Misbegotten Review
Two essay that show zero interest in considering that chiropractic could potentially cause harm, the latter including the argument that it is real doctors that kill people and in comparison out ‘an order of magnitude greater than the side-effects attributed to spinal manipulation.’ See. If you kill small numbers of people, it is not important. Safety only matters when you kill people in large numbers. In the risk/benefit calculus of medicine, an intervention that has no benefit should cause no harm.
If there are concerns in the chiropractic community expressed about these complications, I can’t find them. If there is to improve chiropractic care, I can’t find it. If there is any quality/safety research being done, please rub my nose in it. I would love to know and those involved or aware of such research should trumpet the results for all to read. All the literature and letters to the editor I can find concerns denying there is a problem at all.
At least I could find concerns in the acupuncture literature : ‘A single injury – let alone a fatality – caused by acupuncture is one too many.” Exactly right. If you are practicing prescientific magic, it should have a mortality and morbidity of zero. Again if there is ongoing research into improving the safety of acupuncture, I cannot find it either.
Both seem far more interested in the messenger (Dr. Ernst) and his malevolent intent rather than the message.
It is, I think, a key difference between medicine and its ‘alternatives’. The former takes safety and quality seriously and strives constantly with research and its application to improve care. The alternatives? Nope. Not yet. And probably never.
Addendum. This may be the last post of 2010. I would like to say on behalf of myself and the other authors of SBM (who can disagree in the comments if they desire): Happy New Year. Enjoy 2011 since the world will end in 2012. With one exception, we have a wonderful group of readers and commenters and at times I learn more from the comments that I do preparing my posts. Stay healthy, my friends.

The Miracle Cure for Everything
Steven Novella on October 8, 2014
Link

One common feature of pseudoscience is that proponents of a specific belief tend to exaggerate its scope and implications over time. In the world of physics this can eventually lead to a so-called “theory of everything” – one unifying theory that explains wide-ranging phenomena and displaces many established theories.
In medicine this tendency to exaggerate leads in the direction of the panacea, the miracle cure for everything. Why does this happen?
There are numerous examples. Here is a video of Bruce McBurney trying to sell his Precious Metals Nano Water to investors in the Dragon’s Den. The product is nothing but distilled water with a tiny amount of silver. McBurney claims that this magic water will essentially cure everything, all bacterial and viral infections, and even cancer.
The panacea is also not the sole domain of the lone crank. Straight chiropractors essentially believe that adjusting the spine can cure everything from bed wetting to asthma, and yes, even cancer.
What factors predispose to the panacea claim?
Theory of Everything
The cure for everything is partly related to the theory of everything. If all disease is caused by blockages in the flow of life energy through the spine, then all disease can be cured by adjusting the spine. In the case of Hulda Clark, all disease was allegedly caused by the liver fluke, and so treating that nasty scourge on humanity could therefore cure all disease.
In the era before science, when medical treatments were based on philosophy rather than an empirical understanding of the world, this approach was unavoidable. Galenic medicine believed that health and illness was a matter of the balance among the four humors, and so all conditions could be treated by bloodletting or purging. The Eastern version of this used acupuncture and cupping for bloodletting, which in the early 20th century was reworked as using needles to balance the life force.
Chiropractic and acupuncture have a similar history in that they began as healthcare theories of everything, but in the era of science are desperately clinging to legitimacy by claiming they improve subjective symptoms, those most amenable to placebo effects. Still, they retain their true believers, such as medical acupuncturists who will treat anything with acupuncture.
The scientific approach to health and disease has moved in the opposite direction from the theory of everything. The more we learn about the complexities of the human body, the more we discover all the many different ways that health can be affected. Every aspect of the biological machine can break down or not work optimally, causing illness. Causes may include genetic, traumatic, neoplastic, nutritional, degenerative, infectious, autoimmune, toxic, metabolic, environmental, biochemical, or physiological.
Modern medical theories of everything are maintained in two basic ways. The first is simply to deny modern science and everything that has been discovered about health and disease. This requires a profound level of scientific illiteracy, but this is unfortunately not uncommon.
The second method is to argue that, even though there may be many causes of disease, the body has an unlimited ability to heal itself. One thing keeps the body from perfectly healing itself, and so if you treat that one thing, self-healing will be restored – no matter what the problem. This is, for example, the subluxation theory of chiropractic. Subluxations block the flow of innate intelligence that, unhindered, would heal whatever ails you. This is precisely why alternative practitioners frequently tout that their treatments promote or enhance self healing.
Lack of scientific process
Even without the alternative philosophical underpinnings, there is a tendency for dubious treatments to undergo indication creep over time. A treatment that starts out being used for one specific indication has a growing list of conditions it can treat or cure, even conditions with very different real underlying causes.
This happens because the process that is being used to determine if the treatment works is flawed in the first place. Typically unscientific treatments are based upon anecdotal evidence, which is susceptible to placebo effects. Proponents are not being skeptical, nor are they conducting the kinds of studies that are capable of showing that the treatment does not work.
In fact the process they use is designed to show that the treatment does work. Therefore, no matter what they try it for, it will seem to work. They may naively come to believe that it works for everything. In some cases they may then backfill an explanation for why it works for everything, leading again to the theory of everything.
Essentially, if your beliefs and claims are disconnected from reality by the absence of a skeptical scientific process of investigation, then those beliefs will tend to drift off further and further into fantasy land.
Marketing
The final major factor is the simplest – marketing. If you have a product to sell, you want that product to have as wide a market as possible. In medicine this means as many indications for your treatment as possible. In fact, why limit your market at all? If your treatment works for every indication in every population, then you have maximized your potential customer base.
This does not necessarily mean that those selling panaceas are always knowingly lying, although in some cases that certainly seems to be true. Rather, there is a spectrum. There is a powerful motivation to believe that your treatment has wide-ranging implications. If you discover a treatment that is effective for some cases of athletes foot, that is an achievement and might even be highly profitable. But if you discover the treatment for all infections, or all cancers, or all human disease, then you should become world famous and fabulously wealthy. This is a powerful motivation to believe.
Even legitimate scientists fall prey to the allure of believing their discovery is bigger than it actually was. They have the rest of the scientific community to give them a reality check.
Companies also are highly motivated to exaggerate the indications and effectiveness of their products, and they need effective regulations that require scientific evidence to keep that trend in check.
In the world of alternative medicine and supplements, there is no reality check. The tendency to exaggerate claims is therefore unimpeded.
Conclusion
We have warned often on Science-Based Medicine to beware the “one cure for all disease.” The greater the claims for any treatment, the more improbable those claims become, and the greater should be the level of skepticism.
Biology is complex, and diseases have many causes. It is highly improbable that any one treatment will address a significant portion of human illness. Skepticism should also be high for any intervention that is claimed to address diseases or disorders that seem to have very different causes.
The “self-healing” gambit, while appealing, is also not realistic. Our bodies do have some ability to heal themselves. This ability to fight off infection, heal wounds, and compensate for illness can keep us going for many decades. The various systems of the body, however, break down in many ways, may be overwhelmed by an infection, can suffer trauma, or may have been suboptimal in the first place (such as with a genetic mutation). Entropy always wins out in the end.
We do not have an infinite ability to keep ourselves forever in perfect health. This is a religious belief that runs contrary to overwhelming scientific evidence. It’s a seductive belief, however.
The reality check of science may be disappointing to our emotional desires, but at the same time it has given us the actual ability to prolong life and improve quality of life significantly. I personally would never trade the hard-won knowledge of science for the comforting fictions of the cure-all.

So-Called Alternative Medicine
Edzard Ernst calls it “So-Called Alternative Medicine”. This insider’s view of SCAM is a new book from an prolific researcher and author.
Scott Gavura on June 14, 2018
Link

I have been a fan of Dr. Edzard Ernst for over a decade. His book Trick of Treatment, co-written with Simon Singh, was one of the first books I read on alternative medicine that gave me a perspective of just what “alternative medicine” was (or purported to be). As a pharmacist working occasionally in a drug store that focused on supplements and products like homeopathy, I knew these products lacked evidence of effectiveness, but had never really considered Complementary and Alternative Medicine (CAM) as a cult-like industry, or the double-standard from regulators that enabled all of these sometimes-absurd products to be sold in pharmacies. Dr. Ernst has written several books on alternative medicine, and Harriet Hall has reviewed most of them. Now Dr. Ernst has written SCAM: So-Called Alternative Medicine, a collection of 50 essays on topics related to SCAM. Each chapter is an essay, and each essay examines some aspect of SCAM. This book is not for true believers. If you’re a regular reader of this blog, then this book is for you.
Why “so-called” alternative medicine?
So-called alternative medicine (SCAM) goes by many names such as integrative medicine (which is really more of a brand than a specialty), holistic medicine, unorthodox medicine, and functional medicine. Tim Minchin describes it best, in my opinion. Ernst says that whatever you want to call it, it is not an alternative:
If a therapy does not work, it cannot be an alternative to medicine
If a therapy does work, it does not belong to alternative medicine, but to medicine.
Ernst is a physician and an academic, and his impressive credentials deserve a mention. A German by birth, he earned a medical degree and practiced in a setting where SCAM was fully embedded into conventional medicine. He was a homeopath and even worked in a homeopathic hospital. After spending time in a research position in Austria, he took the (world’s first) professorship in Complementary Medicine at the University of Exeter in England. Ernst built a research team and a research program, determined to critically evaluate SCAM with good science. His team was highly productive, conducting dozens of clinical trials and systematic reviews and compiling, over time, over 1,000 publications. But it was clear relatively early on that:
    • few forms of SCAM seemed to work
    • many seemed not to do more good than harm
    • and most SCAMs were so under-researched that it was impossible to tell
Ernst’s 2001 book, The Desktop Guide to Complementary and Alternative Medicine examined 700 treatment/condition pairs and concluded that only 7.4% were supported by good evidence. Realizing the vast gap between popular perceptions about the efficacy of SCAM and the reality, Ernst broadened his publishing targets from academic publications for health professional to eventually, consumer books like Trick or Treatment. Ernst prefers science over wishful thinking, and was surprised to find that SCAM enthusiasts felt otherwise. Ernst realized this early on in academic posting at Exeter:
On short notice, I had accepted an invitation to address this meeting packed with about 100 proponents of alternative medicine. I felt that their enthusiasm and passion were charming but, no matter whom I talked to, there seemed to be little or no understanding of the role of science in all this. A strange naïvety pervaded this audience: alternative practitioners and their supporters seemed a bit like children playing “doctor and patient”. The language, the rituals and the façade were all more or less in place, but somehow they seemed strangely detached from reality. It felt a bit as though I had landed on a different planet. The delegates passionately wanted to promote alternative medicine, while I, with equal passion and conviction, wanted to conduct good science. The two aims were profoundly different. Nevertheless, I managed to convince myself that they were not irreconcilable, and that we would manage to combine our passions and create something worthwhile, perhaps even groundbreaking.
During my short speech, I summarized my own history as a physician and a scientist and outlined what I intended to do in my new post—nothing concrete yet, merely the general gist. I stressed that my plan was to apply science to this field in order to find out what works and what doesn’t; what is safe and what isn’t. Science, I pointed out, generates progress through asking critical questions and through testing hypotheses. Alternative medicine would either be shown by good science to be of value, or it would turn out to be little more than a passing fad. The endowment of the Laing chair represented an important mile-stone on the way towards the impartial evaluation of alternative medicine, and surely this would be in the best interest of all parties concerned.
My audience, however, was not impressed. When I had finished, there was a stunned, embarrassed silence. Finally someone shouted angrily from the back row: “How did they dare to appoint a doctor to this chair?” I was startled by this question and did not quite understand. What had prompted this reaction? What did this audience expect? Did they think my qualifications were not good enough? Why were they upset by the appointment of a doctor? Who else, in their view, might be better equipped to conduct medical research?
It wasn’t until weeks later that it dawned on me: they had been waiting for someone with a strong commitment to the promotion of alternative medicine. Such a commitment could only come from an alternative practitioner. A doctor personified the establishment, and “alternative” foremost symbolized “anti-establishment”. My little speech had upset them because it confirmed their worst fears of being annexed by “the establishment”. These enthusiasts had hoped for a believer from their own ranks and certainly not for a doctor-scientist to be appointed to the world’s first chair of complementary medicine. They had expected that Exeter University would lend its support to their commercial and ideological interests; they had little understanding of the concept that universities should not be in the business of promoting anything other than high standards.
The content
Ernst divides his essays into several categories:
    • The Basics: What is SCAM? What is EBM? How much of EBM is actually evidence-based?
    • Common Problems in SCAM: The “root cause”, fear mongering, holism, conspiracy theories, and quackademia
    • Research and Researchers: Research in SCAM, promotion masquerading as research, NCCAIM, how to tell good from bad research
    • Healthcare Practitioners: Inside the brain of a SCAM practitioner, Doctors of Integrative Medicine, Veterinary Medicine, Nursing, Midwifery, Pharmacy
    • Patients and Consumers: Patient choice, when orthodox medicine has nothing more to offer, the wellness mania, alternative cancer cures
    • The Funny Side: Several humorous essays including How to become a Charlatan
Many SCAM advocates have accused Ernst of having a conflict of interest. He agrees he has a conflict, but not in the way his opponents claim. He notes:
    • evidence is a prerequisite for progress in healthcare,
    • evidence must be established by rigorous research,
    • we should not tolerate double standards in healthcare,
    • patients deserve to be treated with the best available treatments,
    • making therapeutic claims that are not supported by sound evidence is wrong.
Why SCAM research is rarely negative
Some SCAM journals and SCAM researchers never publish negative findings. This isn’t evidence of SCAM’s effect, but rather it’s the consequence of a trial design that cannot be negative. Ernst calls this the A vs. A+B design, and he describes it in a blog post studying a trial that purported to show benefit from acupuncture on cancer-related fatigue. Patients received “usual care” or “usual care” plus acupuncture. The acupuncture group did better. Ernst notes this design is designed to show an effect:
Imagine you have an amount of money A and your friend owns the same sum plus another amount B. Who has more money? Simple, it is, of course your friend: A+B will always be more than A [unless B is a negative amount]. For the same reason, such “pragmatic” trials will always generate positive results [unless the treatment in question does actual harm]. Treatment as usual plus acupuncture is more than treatment as usual, and the former is therefore more than likely to produce a better result. This will be true, even if acupuncture is no more than a placebo – after all, a placebo is more than nothing, and the placebo effect will impact on the outcome, particularly if we are dealing with a highly subjective symptom such as fatigue.
The Tricks of the SCAM Trade
One of my favourite essays describes some of the tactics used by SCAM advocates to promote their services.
Treat a non-existent condition: There’s nothing better than treating a condition the patient doesn’t even have. From the chiropractor’s subluxation to the acupuncturist’s chi blockage, these diagnoses are made up by the practitioner, and have no basis in reality.
Maintenance treatment: Persuade the patient who is healthy that they need “maintenance” treatments to stay healthy, such as IV vitamin infusions or the chiropractor recommending regular adjustments for spinal alignment. To the worried well, this may sound attractive, but it isn’t based on any objective benefit.
Things must get worse before they get better: Patients may feel nothing, or even worse, after a SCAM treatment. Practitioners may describe this as a “healing crisis”, an imaginary phenomena. In the case of conditions that are chronic or wax and wane, the practitioner can attribute worsening to the “healing crisis” and claim any improvement as evidence of effectiveness.
A cure takes a long time: To continue SCAM treatments despite a lack of any improvement is hard to justify on medical grounds, but practitioners make this claim routinely.
The problem is due to conventional medicine: With a nod to the conspiracy theories that are endemic to SCAM, problems are often blamed on conventional medicine. You see this routinely in harms or circumstances attributed to therapies like vaccines.
Detox: A marketing slogan and not an actual treatment, CAM proponents claim detox will rid the body of non-specified “toxins” that conventional medicine fails to acknowledge or treat.
The test of time: Some SCAMs have been around for hundreds of years (or more). SCAM advocates point to this long history of use as evidence of safety and effectiveness. Ernst points to bloodletting as a treatment with a long history of use that was appropriately discarded when the human body became better understood.
Energy: Energy is one of those terms that means whatever the SCAM practitioner wants it to mean, but it’s usually a proxy for “vital force“. Energy just sounds more scientific. While vitalism has been discarded by science-based medicine, it’s at the root of many SCAMs, like naturopathy.
Stimulating the immune system: Demonstrating rather startling ignorance of immunity, SCAM “stimulation” is a placebo. Moreover, it’s probably something we wouldn’t want to do, if we could.
A good book for the fledgling skeptic or SBM enthusiast
If you read SBM daily and are familiar with Dr. Ernst’s work or his blog, this book covers familiar topics. As a non-academic, I enjoyed this book because it made me consider aspects of SCAM from the perspective of a researcher. Despite Ernst’s deep knowledge of this topic, he writes in an easily approachable style – this is not dense medical writing. This book is not likely to change the minds of any true believers – but nor is that the audience. If you’re interested in the role of science in medicine, and on how to sort out good from bad when it comes to health and healthcare decision-making, Ernst’s book is highly recommended.

[Full disclosure: Dr. Ernst’s publisher provided me with a free copy of his book.]

March Madness: Basketball, Brackets and Psi! Oh my!
Clay Jones on April 10, 2015
Link


In 2011, psychologist Daryl Bem published a highly controversial series of nine experiments designed to tease out the potential existence of precognition, the ability to experience future events. In order to isolate the potential influence of future events on the present, Bem’s experimental design reversed the standard order of psychological investigations. In one experiment, for example, subjects were allowed to practice with random words after having already taken a memory test comprised of some of them.
Bem’s results were controversial for many reasons, mostly because they were positive. But the question of whether or not they should have even been published in the first place, regardless of the results, was raised by many in the scientific and skeptical community. As history has shown us repeatedly, the risk of falsely legitimizing nonsense by publishing positive but typically poorly-designed studies is very real. And negative studies tend to be ignored by believers and policy makers.
Bem’s experimental design was made available from the beginning of his research in 2002, and he encouraged others to perform their own studies. Unfortunately for Bem, attempts at replicating his findings were largely reported, if not always published, as negative. Apparently this was not enough to discourage the intrepid believer. In fact, he currently has a meta-analysis of 90 experiments using the same protocols as his original research under editorial review.
The conclusions of his new analysis [PDF] are, not surprisingly, supportive of the existence of precognition, and in his manuscript he attempts to address all of the skepticism regarding his findings. He even quotes Feynman and implies that physicists are taking the possibility of quantum-based explanations of psi seriously. It’s an interesting read to say the least.
What does NCAA basketball have to do with the topic of psi research?
A new name in the field of parapsychology research has been making headlines since going public with his efforts on the 1st of the month, and not surprisingly the media coverage is increasing now that his findings have been announced. Dr. Mort Fishman, a psychologist and paranormal researcher out of Tuono di Legno University in Florence, has ingeniously incorporated one of the world’s most popular sporting events, the NCAA Men’s Basketball Tournament, into a study of precognitive abilities. The results, if they are to be believed, may change the minds of even the hardiest psi skeptics.
In anticipation of March Madness each year, millions of basketball fans around the world fill out tournament brackets in an attempt to correctly predict which teams will win, and ultimately who will emerge victorious from the championship game. Many make a contest out of it, with office pools now a nearly ubiquitous aspect of life this time of year. ESPN even holds a Tournament Challenge, the winner of which is given a chance for $20,000 and a trip to Hawaii. It might even be Mitt Romney this year, leading some to regret their vote in the 2012 Presidential election.
The reasons people give for their choices are myriad, ranging from a thorough evaluation of the various teams’ stats to something as primitive and powerful as a gut feeling or how cute the mascot is. Fishman, an avid basketball fan who played three seasons for the European Basketball Federation’s Florence Nightingales, set out at the beginning of the NCAA college basketball season to determine if there might be a precognitive component to these choices. Inspired by Bem’s study designs, he also incorporated a reversal of standard protocols into his research.
The typical approach to filling out a bracket is to do so before the first game of the tournament. In fact, most official contests require it in order to avoid any unfair advantage. Fishman, who had a league record foul-to-rebound ratio of 2.5 during his stint on the Nightingales, instead asked study participants to fill out their bracket after the completion of the championship game, which was won by beloved perennial tournament underdogs, the Duke Blue Devils. According to Fishman, the most challenging aspect of the study was preventing participants from being exposed to any potential contamination with information about the teams during the season.
We went to great expense to protect subjects from potential bias or even outright cheating by basing their bracket picks on outside information. They were sequestered in a reasonably priced hotel, without access to the internet, television, or radio from November 1st of last year through Monday night. They were not allowed to speak to anyone outside of each other and us. No family. No friends. I think there were a few old Reader’s Digests and maybe a Redbook in the room. Progress requires sacrifice.
Once the brackets were completed and turned over to the research team, the real science began. Each subject entered a sensory deprivation tank and was randomly exposed to the results of 25 of the tournament’s 67 total games. The brackets were analyzed using math to see if subject picks were retroactively impacted by knowing the result of games after the fact. Choosing to courageously forego the stifling process of peer review, Fishman announced the positive findings at a press conference held in the parking lot of a Buffalo Wild Wings eight minutes from Lucas Oil Stadium on East Washington Street in Indianapolis.
Conclusion
As believers in Science-Based Medicine, we must have open minds when it comes to fringe or even highly implausible claims because you never know when seemingly-incontrovertible assumptions about the natural world will be proven wrong. Science is imperfect and historically has been very wrong on many occasions. Experts used to believe that the Earth was flat and we still don’t know how Tylenol even works.
Intellectual humility is a key component of scientific skepticism. Sometimes ideas which were ridiculed turn out to be true. Having a closed mind might help prevent any untrue new idea from being accepted, but some of them are pretty interesting and already very popular with the public. Who are we to judge truth from fiction?
Though the existence of psi has now been proven by the Fishman study, it doesn’t mean that more studies aren’t needed. In fact, the hard part lies ahead. We need to figure out how best to harness its potential in order to improve patient outcomes, and only science can lead the way in that regard. But we must not be afraid of the unknown or of different approaches to discovery. As Endocrinologist Dr. Deepak Chopra said, “All great changes are preceded by chaos.”

Persecution of Scientists Whose Findings Are Perceived As Politically Incorrect
Alice Dreger has written a delightful, breezy, and slightly terrifying book on activism gone bad, and how it impacted scientists just trying to uncover facts and truth. Highly recommended!
Harriet Hall on February 16, 2016
Tweet

It dates back at least to Galileo. A scientist finds evidence that contradicts a cherished popular belief. Instead of a rational examination of his evidence, he is subjected to vicious personal attacks. Alice Dreger examines the phenomenon in her book Galileo’s Middle Finger: Heretics, Activists, and the Search for Justice in Science. She is eminently qualified to do so. She is a professor of clinical medical humanities and bioethics, a historian, a gifted writer, an activist for patient rights, and an indefatigable investigative journalist who has herself been a victim of the kind of persecution she describes.
The histories she recounts are horrifying. She gives example after example of activists using lies and personal attacks to suppress evidence they don’t like. She reveals dirty linen in the most unexpected places.
Bailey and transgender
A subset of the LGBT community became incensed when sex researcher Michael Bailey suggested that transgender is more complicated than just a matter of being born with the body of one sex and the brain of another, that transgender identity is not just about core inborn gender, but it also involves sexual orientation and erotic feelings. They mounted a sophisticated campaign to bury that idea by killing the messenger (the specifics are discussed in exhaustive and fascinating detail in a 2008 article by Dreger). Led by a trio of powerful transgender women, they falsely accused him of serious crimes including abusing the rights of subjects, having sex with a transsexual research subject, and making up data. They even insinuated that he had been sodomizing his own children. They nearly ruined his career and his life.
Dreger was already an activist for the rights of intersex children, fighting to prevent children born with ambiguous genitalia from being surgically altered just to satisfy the prejudices of adults. She advocated waiting and allowing them to make that decision for themselves when they reach maturity. She had been working with the transgender community, and she became embroiled in the controversy over Bailey’s findings. Her meticulous research uncovered facts that directly contradicted the accusations; for instance, on the date Bailey was supposed to have had sex with the transgender subject, there was clear proof that he had been in a different city. The whole campaign against him was an obvious setup. When she published her findings, she was portrayed as an enemy of intersex people and an enemy of all LGBT people. Lies were told about her, for instance that she was driven out of a job for bad behavior, when she had voluntarily changed jobs for purely personal reasons. The animosity and venom reached incredible heights.
A repeating pattern
A pattern repeats itself. The scientist’s message is distorted; critics often don’t bother to read the book or the article in question, but are content to rely on someone else’s garbled misinterpretation. They are not angry about what the scientist actually said, but about what they think he or she said. When E. O. Wilson wrote his book Sociobiology, his intent was to help humans understand their own nature. His message was misrepresented as an attempt to exonerate people from responsibility for crimes or social problems, and even as supporting “the eugenics policies which led to the establishment of gas chambers in Nazi Germany.” Hatred of Wilson escalated to the point that a group of his detractors rushed onstage and doused him with a pitcher of water at a scientific conference.
The persecution of Chagnon
Napoleon Chagnon is a cultural anthropologist who studied the Yanomamo tribe in the Amazon, living with them for long periods. In contradiction to the anodyne image of the peaceful, noble savage, he found “the fierce people.” Males fought wars and fought violently over fertile females; there was domestic brutality, ritualized drug use, and ecological indifference. This was a message that disturbed a lot of people. Instead of examining his findings, they resorted to personal attacks. A journalist, Patrick Tierney, wrote a book accusing Chagnon of atrocities against the Yanomamo people, such as deliberately starting wars and causing a measles epidemic with a bad vaccine and then withholding medical care. The American Anthropological Association appointed a task force to investigate, but it did such a poor job that its findings later had to be rescinded. Chagnon’s reputation was ruined.
A little basic fact checking would have put paid to most of Tierney’s accusations, as Dreger was able to demonstrate with her own investigation many years later. Tierney cited a dossier by Leda Martin as evidence for the charges against Chagnon. Dreger contacted Martin and learned that Tierney wrote the dossier himself, gave it to Martin to translate, and then pretended Martin was the author. One of the most damning pieces of evidence related to a doctor who had worked with Chagnon during the measles epidemic. Actually, it was no more than hearsay. It was what the doctor’s son said his father had told him in a private conversation. The doctor had since died, but Dreger contacted the doctor’s son, who now believes his father made up the story to make himself look good, something he had a pattern of doing on other occasions.
Childhood sexual abuse
Bruce Rind and his colleagues published a meta-analysis of studies of college students who had been targets of sexual advances by adults when they were children. They analyzed which factors were more likely to be associated with long-term psychological harm (female sex, incest), and suggested that it didn’t make sense to lump together (a) molestation of a five-year-old by a sixty-year-old and (b) consensual sex between a sixteen-year-old and a twenty-year old. Both were categorized as childhood sexual abuse (CSA), but the former was more likely to result in long-term psychological harm than the latter.
They found that in a surprising proportion of cases, CSA doesn’t result in lasting harm. This should have been good news, reassuring evidence of human resilience. Instead, they were accused of promoting pedophilia! The Rind study aroused such outrage that its scientific findings were condemned by an act of Congress, the only time in history that has ever happened.
Richard Dawkins has recently been vilified for saying essentially the same thing, that the kind of “mild pedophilia” he experienced as a child in an English boarding school did not cause lasting harm. He tried to put that kind of abuse into historical perspective, saying it was not condemned at the time the way it would be condemned today. His words were interpreted by his critics to mean that he condones pedophilia, the exact opposite of what he said: “by the same standards as I or anyone would [condemn it] today.” On Twitter, Dawkins responded, “Mild pedophilia is bad. Violent pedophilia is worse. If you think that’s an endorsement of mild pedophilia, go away and learn how to think.” By analogy, theft is always wrong, and the law universally condemns and punishes it; but the law recognizes that grand theft is worse than petty theft. Bernie Madoff got a harsher punishment than an adolescent with a first-time shoplifting offense, but that doesn’t constitute an endorsement of shoplifting.
More examples
Similar outrage followed the finding that rape has a sexual component and is not merely an expression of violence. Which makes sense: if there were no sexual component, rapists would be just as likely to choose some other form of violence, and no rapist would ever be able to sustain an erection and impregnate a victim. The scientists involved got death threats.
Elizabeth Loftus got into very deep trouble for finding that human memory is fallible and that some allegations of child sexual abuse were based on false “recovered repressed memories” of events that never happened. She was accused of denying the possibility of child sexual abuse and attacking its victims, something she absolutely did not do. She was even sued over what she wrote, but her work was eventually vindicated.
Unethical treatment for CAH
Dreger’s latest activism has to do with congenital adrenal hyperplasia (CAH), a condition where exposure to excess male hormones in utero can lead to masculinized genitalia in a female child. Genetic testing can identify who is at risk; and a pediatric endocrinologist named Maria New, among others, has been treating those women with the steroid dexamethasone to prevent intersex anomalies. Only female fetuses are affected, but the treatment often has to be started even before the sex of the fetus is known. Since the gene is recessive, only one in four female fetuses will be affected, and there is considerable variation in how much they are affected, ranging from a nearly typical female to a nearly typical male appearance. There has been only one published case report of an assumed successful treatment with dexamethasone. No controlled studies. So prenatal use of dexamethasone for CAH is an off-label use of a drug that has not been shown to work, and must be considered unethical outside of controlled studies. Dreger’s investigations found a host of problems: violation of research standards, lack of informed consent, misrepresentations, lack of follow-up, and even direct evidence that the treatment can be harmful. Adverse effects include growth disorders, mood disorders, and mental retardation; and boys accidentally exposed to dexamethasone may be genitally and behaviorally feminized. A trial in Sweden was shut down because of these concerns, and the researchers said they would no longer provide the intervention even in the context of a prospective clinical trial.
Conclusion: Good intentions, bad results, worse reporting
Good people with the best intentions can do bad things. Dreger concludes that we need both scholars and activists; we need people pushing for truth and for justice if we’re going to get both right. She is fair to both sides: she exposes the sins of activists and the sins of scientists and scientific institutions. Highly respected thinkers like Jared Diamond, Edward O. Wilson, Elizabeth Loftus, and Steven Pinker have endorsed the book. Diamond says it would be “this year’s most gripping novel” except that her stories are true. Wilson says she “reveals the shocking extent to which some disciplines have been infested by mountebanks, poseurs and even worse, political activists who put ideology ahead of science.” A good read, valuable information, a cautionary tale, and plenty of food for thought.

Keeping the customer satisfied
These days patient satisfaction is becoming more and more important in judging how well hospitals and physicians are doing. Indeed, Medicare reimbursements are linked in part to a patient satisfaction survey administered by the government. The underlying assumption is that patient satisfaction correlates with high quality care, But is that true? I have my doubts. Indeed, there is evidence that, at best, patient satisfaction correlates poorly with quality medical care and, at worst, that it negatively correlates with quality.
David Gorski on March 19, 2012
Link
One thing about blogging once a week or so compared to my other blogging gig, which is usually close to every day, occasionally more often, is that I really can’t cover everything I want to cover for this blog. Even more so than at my not-so-super-secret other blogging gig, I have to pass on topics that could be fodder for what could be excellent to even awesome posts—or, self-congratulating hyperbole aside, at least reasonably interesting to the readers of this blog. When that happens, I can only hope that one of my co-bloggers picks up on it and gives the subject matter the treatment it cries out for. Or, sometimes, such subject matter just has to be dealth with elsewhere by me—or not at all. Even a hypercaffeinated blogger like myself has limits.
Sometimes, however, I actually get a second chance. In other words, I get a chance to revisit a topic that I passed by. Usually, this happens when something new happens that gives me an excuse to revisit the topic. So it was last of week, when I was perusing the New York Times by an oncology nurse named Theresa Brown. Her article was titled, appropriately enough, Hospitals Aren’t Hotels. It will become very apparent very quickly why in a moment. But first, let’s sample Brown’s article a bit, because it brings up an issue that is very pertinent to science-based medicine:
“You should never do this procedure without pain medicine,” the senior surgeon told a resident. “This is one of the most painful things we do.”
She wasn’t scolding, just firm, and she was telling the truth. The patient needed pleurodesis, a treatment that involves abrading the lining of the lungs in an attempt to stop fluid from collecting there. A tube inserted between the two layers of protective lung tissue drains the liquid, and then an irritant is slowly injected back into the tube. The tissue becomes inflamed and sticks together, the idea being that fluid cannot accumulate where there’s no space.
I have watched patients go through pleurodesis, and even with pain medication, they suffer. We injure them in this controlled, short-term way to prevent long-term recurrence of a much more serious problem: fluid around the lungs makes it very hard to breathe.
A lot of what we do in medicine, and especially in modern hospital care, adheres to this same formulation. We hurt people because it’s the only way we know to make them better. This is the nature of our work, which is why the growing focus on measuring “patient satisfaction” as a way to judge the quality of a hospital’s care is worrisomely off the mark.
As a surgical resident, I rotated on the thoracic surgery service on multiple occasions over the course of my five clinical years. As part of my duties on that service, I’d see all the chest tube and pleurodesis consults, put in most, if not all, of the chest tubes, and do most of the pleurodesis procedures, at first supervised by the cardiothoracic fellow and then on my own. They’re not difficult procedures (I had learned to insert chest tubes when I was an intern on the trauma service) and learning to do pleurodesis wasn’t difficult either. I’m not sure if pleurodesis is the most painful procedure I did, but it certainly wasn’t pleasant. Think of the procedure this way: The goal of the procedure is to suck out all of the pleural effusion (a collection of fluid surrounding the lung) and then to get the pleura lining the lung to stick to the pleura lining the inside of the chest wall by, in essence, “roughing them up” so that they become very inflamed and stick together when they heal. The “irritant” (as Brown puts it) that we used to use was nothing more fancy than a slurry of sterile talc, although there are other irritants that can be used, such as bleomycin, tetracycline, povidone iodine.
The point, of course, is that we as doctors sometimes have to be, as Nick Lowe once put it, “cruel to be kind in the right measure.” As a surgeon, I’m acutely aware of this necessity. Surgery hurts. There’s just no way to get around it. The best we can do is to try to minimize the pain we cause by slicing people open and forcibly rearranging or removing parts of their anatomy for (hopefully) therapeutic intent; we can’t eliminate the pain. But even surgery isn’t the worst that we as physicians inflict upon patients in the name of trying to heal them. For examples, my colleagues in medical oncology administer highly toxic chemicals to patients, chemicals that make their hair fall out, temporarily weaken their immune systems, rendering them susceptible to life-threatening infections, cause neuropathy, and produce all sorts of other adverse effects. Think of bone marrow transplants with stem cell rescue. It’s a procedure in which doctors literally destroy the patient’s existing bone marrow (and thus the vast majority of his immune system) and then reconstitute it using either the patient’s own hematopoietic stem cells or marrow obtained from a donor. It’s unpleasant, takes weeks, and puts the patient at risk for death from the procedure and, in the case when the marrow used is from a donor instead of the patient, puts the patient at risk for graft versus host disease.
We as physicians don’t do these things to patients because we like to cause suffering. We do them because what science tells us about the diseases being treated also tells us that these are the sorts of things we have to do to save the lives of patients with serious diseases. In the case of cancer, for instance, gentler interventions just don’t work as well. Contrary to how some quackery propagandists like to portray physicians, we don’t do these things because we get off on it. We do them because the benefits outweigh the risks, and because we can save lives. Meanwhile, we are continually doing research to find treatments that are more efficacious, as well as less risky and unpleasant. It might well be that doctors in the far future will recoil in horror at our current treatments, much the way Dr. McCoy did when he encountered dialysis and chemotherapy, but they’re the best that we have now, and we are working to improve them.
Brown is correct when she points out that these days patient satisfaction is becoming more and more important in judging how well hospitals and physicians are doing. She also points out that by October 2012—yes, this year—Medicare reimbursements will be linked in part to a patient satisfaction survey administered by the government known as the Hospital Consumer Assessment of Healthcare Providers and Systems survey. While the survey itself, as Brown also points out, measures aspects of health care that are important, such as communication between the physicians, nurses, and staff and the patient, how well the patient was educated about his condition, and how clear discharge instructions were, note the underlying assumption behind such surveys is that patient satisfaction correlates with high quality care, But is that true? Brown has her doubts, as do I:
These are important questions. But implied in the proposal is a troubling misapprehension of how unpleasant a lot of actual health care is. The survey measures the “patient experience of care” to generate information important to “consumers.” Put colloquially, it evaluates hospital patients’ level of satisfaction.
The problem with this metric is that a lot of hospital care is, like pleurodesis, invasive, painful and even dehumanizing. Surgery leaves incisional pain as well as internal hurts from the removal of a gallbladder or tumor, or the repair of a broken bone. Chemotherapy weakens the immune system. We might like to say it shouldn’t be, but physical pain, and its concomitant emotional suffering, tend to be inseparable from standard care.
Certainly, good communication, for instance, is essential to patient care and it’s not unreasonable to think that good communication will tend to lead to more satisfied patients. Brown points out that it “ain’t necessarily so” that these sorts of metrics correlate with outcomes, and I’ll show you as study that shows why it ain’t necessarily so. I seriously wonder whether part of what’s fueling the rise of “integrative medicine,” formerly known as “complementary and alternative medicine” (CAM), formerly known as “alternative medicine,” is more than just credulity on the part of physicians, but the result of a successful effort on the part of CAM practitioners and believers to portray CAM as being somehow more responsive to patients than “conventional” medicine. I discussed this in a different context when observing how CAM has tried to co-opt the brand of “patient-centered” care, corrupting it to become equivalent to giving the patient what he, even if what the patient wants isn’t necessarily in his best interest. Clearly, at the very least there is synergy between the rise of the cult of patient satisfaction as the be-all and end-all of quality metrics and the rise of CAM as a means of increasing patient satisfaction without evidence for improving outcomes.
In any case, if there’s one thing that practitioners and promoters of integrative medicine and hospital administrators seem to agree on, it’s that “patient satisfaction” is very, very important. In the conventional medical world, it’s become so important that hospital administrators live and die by patient satisfaction surveys. In particular, they live and die by a survey that makes the HCAHPS seem fairly reasonable (it does, after all, pay particular attention to communication and the clarity of discharge instructions), namely an instrument known as the Press-Ganey survey. In fact, Press-Ganey itself sells its services as “driving performance excellence” in health care. The inherent assumption is that if patients are satisfied then the hospital and health care providers must be doing a good job. But it’s subtler than that. The underlying assumption is actually that patient satisfaction equals quality, and the further assumption is that Press-Ganey scores reflect patient satisfaction. Never mind that Press-Ganey scores include questions about a whole host of things that have nothing to do with the quality of care. For example, parking was a huge problem at the cancer center where I worked at my previous job, and Press-Ganey scores have always taken a hit because parking is a big issue in the surveys. Does that mean we delivered inferior care there? I don’t think so. Both cancer centers I’ve worked for since finishing my fellowship are among the elite 41 cancer centers designated as Comprehensive Cancer Centers by the National Cancer Institute. As far as cancer goes, these are the best of the best.
Similarly, promoters of CAM/IM seem to believe in patient satisfaction über alles. In fact, two large surveys of the state of “integrative medicine” in the U.S. have been published in the last six months or so, one by the Samueli Institute and one by the sugar daddy of quackademic medicine, the Bravewell Collaborative. Did either of them look at actual hard patient outcomes? Well, not exactly. If you’re talking about actual medical outcomes, as in outcomes research, the answer is a resounding no, although incongruously the Samueli survey noted that 71% of hospitals that started integrative medicine programs did so for reasons of “clinical effectiveness.” At least, that’s what they said. How they would know, I have no idea given the lack of outcomes research. Neither survey even bothered with outcomes, other than the risible question asked in the Bravewell survey about which areas each integrative medicine center director thought integrative medicine had the best “success.” As I pointed out at the time, this was a question so incredibly vague that, unless the patient dropped dead immediately upon contact with the therapy recommended, centers could claim some level of “success” with almost anything, and they did. No metrics for what constitutes “success” were described. Instead, the survey just took the word of integrative medical center directors for it regarding their responses to a question about what conditions they considered to be conditions “among their top five most successfully treated conditions.” Not surprisingly, Samueli noted that, while 85% of the integrative medicine centers it surveyed will use patient satisfaction as a metric to evaluate their CAM programs, only 42% plan on evaluating health outcomes and only 31% evaluate quality.
That’s why, if you’re talking about “outcomes” as in patient satisfaction outcomes, then the answer is yes. Both “integrative medical center” surveys focused like a laser beam on patient satisfaction. Lacking any concrete measures for the quality of care they provide, apparently CAM/IM promoters bragged about how happy their patients are with their services. Of course, this makes perfect sense, given that both surveys were nothing more than one huge exercise in argumentum ad populum. Trying to argue that people are very happy with your service is part and parcel of that. Certainly the CAMsters aren’t trying ot argue for the superiority of their woo based on science. Think of it this way. Lots of people are “very satisfied” with the astrologers, psychic mediums, and faith healers that they use. That doesn’t mean that astrology is a science, that John Edward can talk to the dead, or that John of God can use the healing power of Jesus to cure what ails you.
So, both promoters of “integrative” medicine and a large segment of conventional medicine view patient satisfaction as being a major indicator (but, in all fairness, not the only indicator) of quality care. But, again, is this assumption valid? Does patient satisfaction correlate with high quality care? You might be surprised at the answer suggested by a published this February in the Archives of Internal Medicine from a group out of UC-Davis entitled The Cost of Satisfaction: A National Study of Patient Satisfaction, Health Care Utilization, Expenditures, and Mortality.
This study was designed to look for correlations between patient satisfaction and outcomes, asking the question: Is there a correlation between health care outcomes and patient satisfaction? The authors conclude that the answer is yes, but not in the way that you’d think. In fact, if this study is to be believed, if anything increased patient satisfaction is correlated with worse outcomes in at least some measures. Let’s go to the tape (or the peer-reviewed study).
The authors frame the question thusly:
Satisfied patients are more adherent to physician recommendations and more loyal to physicians, but research suggests a tenuous link between patient satisfaction and health care quality and outcomes. Among a vulnerable older population, patient satisfaction had no association with the technical quality of geriatric care,8 and evidence suggests that satisfaction has little or no correlation with Health Plan Employer Data and Information Set quality metrics.
In addition, patients often request discretionary services that are of little or no medical benefit, and physicians frequently accede to these requests, which is associated with higher patient satisfaction. Physicians whose compensation is more strongly linked with patient satisfaction are more likely to deliver discretionary services, such as advanced imaging for acute low back pain.
In order to investigate the relationship between patient satisfaction and outcomes, the investigators undertook a prospective cohort study. Basically, they looked at respondents to the MEPS from 2000 to 2007. The MEPS is described thusly:
The MEPS is an annual nationally representative survey of the US civilian noninstitutionalized population assessing access to, use of, and costs associated with medical services. The MEPS household component uses an overlapping panel design in which individuals are interviewed successively during 2 years. During each year, respondents complete self-administered questionnaires about health status and their experiences with health care. The MEPS sampling frame is drawn from respondents to the National Health Interview Survey, an annual in-person household survey conducted by the National Center for Health Statistics. The National Health Interview Survey data are linked with death certificate data from the National Death Index, enabling mortality ascertainment among MEPS participants.
In brief, the investigators followed over 50,000 adults and linked their questionnaire responses to their mortality outcomes. The results were both good and bad, although, quite honestly, from my perspective they were mostly bad. Let’s start with the good. One correlation that was noted was that patients with higher levels of satisfaction with their care tended to use the emergency room less. It wasn’t a huge amount less, though. In fact, the adjusted odds ratio found by the authors was only 0.92, which means that the patients who had the highest level of satisfaction (the highest quartile on the survey) were not that much less likely to use the emergency room during the study period than those with the lowest level of satisfaction (the lowest quartile). All in all, not that impressively different, but it definitely has to be acknowledged as a positive.
Now let’s look at the negatives, which to my mind are much more negative.
Patients in the study who demonstrated the highest level of satisfaction were more likely to have an inpatient admission (adjusted odds ratio 1.12) than those with the lowest levels of satisfaction, again not that huge a difference. They did, however, account for 8.8% more health care expenditures, including greater prescription drug expenditures. Worst of all, they demonstrated a higher mortality, with an odds ratio of 1.26.
Data like this always have to make you wonder. Is there a confounding variable that accounts for the negative correlation between patient satisfaction and outcomes? And it’s certainly possible that there may be. However, even if there is, at the very least this study is pretty strong evidence that there isn’t much, if any, correlation between patient satisfaction and the actual quality of care as measured by overall mortality results. Why might this be? The authors provide some possible explanations in the discussion. One aspect of this relationship is that patient satisfaction does correlate with how much the physician fulfills the patient’s wishes and expectiations:
Patients typically bring expectations to medical encounters, often making specific requests of physicians, and satisfaction correlates with the extent to which physicians fulfill patient expectations. Patient requests have also been shown to have a powerful influence on physician prescribing behavior, and our findings suggest that patient satisfaction may be particularly strongly linked with prescription drug expenditures.
In other words, giving the patient what he or she wants isn’t always what’s best for the patient. As “paternalistic,” as this might sound, this is not a new observation. Physicians have known this for a very long time. Perhaps the most striking example of this phenomenon is antivaccine parents. Such parents don’t want their children to be vaccinated, but not vaccinating is rarely in the best interests of the child. Physicians who just go along with such parents, such as antivaccine apologist Dr. Jay Gordon, are very popular and likely generate high Press-Ganey scores because they basically give the people what they want. In contrast, pediatricians who try to do the right thing and persuade such parents to vaccinate their children (or even fire such patients whose parents won’t vaccinate) don’t and as a result generate a lot less patient satisfaction. This is an intentionally chosen extreme example, but the same sort of dynamic occurs in more subtle ways in every patient encounter. A less extreme example is very common in primary care, specifically the example of the patient who demands antibiotics for a viral infection. The doctor who acquiesces will have the more satisfied patient than the doctor who does not. But who provided the better care? Not the doctor who gave unnecessary antibiotics, which can select for resistant organisms and cause complications for no benefit.
In an accompanying editorial, Dr. Brenda Sirovich notes that patients very much like early detection and aggressive intervention, even when that intervention might not necessarily be helping them. She tells the story of “A Healthy Man’s Nightmare,” which was an article published in the New Yorker recounting how literature professor Joseph Epstein went from thinking himself healthy at age 60 to surviving a coronary artery bypass surgery. It all started based on a “routine” physical that revealed a low high density lipoprotein cholesterol level, which in turn led directly to a stress test, which in turn led to…well, you get the idea. The result was that Epstein, at 62, considers himself “weakened, with a lasting sense of vulnerability that he eloquently labels ‘heart-consciousness.'” Did aggressive screening help or harm Epstein? It’s not clear. However, Epstein considers himself “lucky” and attributes his good fortune to his physicians, whom he describes as “paragons of excellence.” This anecdote leads Dr. Sirovich to speculate:
Regardless of whether one believes Mr Epstein to have been ultimately helped or harmed by his screening stress test, his satisfaction with the experience should perhaps not be as surprising as I initially found it. Satisfaction with seemingly adverse outcomes of potentially excessive medical care appears to be the norm. Numerous studies have found that patients are consistently highly satisfied with one of the most common downsides of medical care— false-positive test results and the downstream events that follow.5,6 Moreover, such patients are more likely to undergo the same (and likely other) testing in the future, dismissing their anxiety and other adverse effects as a negligible price for a good outcome.
In other words, many, if not most, patients tend to like aggressive intervention, and they tend not to like “watchful waiting.” They want to do something. That’s part of what’s driving the whole controversy over screening, be it screening for cancer, heart disease, or any of the other conditions we routinely screen for. Screening is a complex equation in which balancing risks and benefits is anything but simple. Overaggressive screening can lead to overdiagnosis and overtreatment whose harm outweighs the benefit in terms of lives saved by early detection and intervention. Moreover, the pressure isn’t just from the patient or on the patient. There are what Sirovich refers to as “positive feedback loops” that pressure doctors, too:
The same heuristic operates on the physician. Ransohoff et al7 proposed, a decade ago, that prostatespecific antigen (PSA) screening for prostate cancer exemplifies a system without negative feedback. Regardless of the true net effect (beneficial or harmful) of screening, a physician ordering a screening PSA receives a favorable result: he can reassure the patient with a normal PSA result; celebrate with the patient who has overcome a false positive; or (most compelling for the physician) offer potentially life-saving treatment to the patient whose prostate cancer was “caught early”— notwithstanding the likelihood that the patient’s outcome may be worse because of early detection. Regardless, the physician can feel satisfied, and more certain that ordering the next screening PSA will be the right decision, which will then appear to be the case, and so on.
Positive feedback systems abound in health care, for both physicians and patients. Diagnostically, almost any unnecessary, or discretionary, test (particularly imaging) has a good chance of detecting an abnormality. Acting on that abnormality has an excellent chance of producing a favorable outcome (because a good outcome was already highly likely). Having obtained an excellent outcome, ostensibly owing to a test that was seemingly unnecessary, a natural reaction would be thereafter to perform (or, for patients, undergo) even more discretionary testing in patients with an increasingly negligible likelihood of benefit—and greater risk of net harm.
So, on both sides of the equation, the patient’s and the physician’s, there are many apparent rewards for delivering more care and few disincentives for doing so, at least on the level of the individual patient. It takes outcomes research and randomized studies to determine whether providing “more” care actually does what it is intended to do, how likely it is to benefit each patient, and how likely it is to harm each patient. Sirovich notes that she still thinks there’s an unidentified confounder in this study, given that the excess mortality far exceeded the excess rate of emergency room utilization in the most “satisifed” quartile, but she also points out that this result is plausible, based on what we know already. I agree, which leads me to a bit of a stray thought that I’d like to see discussed in the comments.
That stray thought is that maybe the popularity of CAM is arising from this same impulse, both on the part of physicians and the part of patients. For example, patients, faced with conditions for which standard science-based medicine has little to offer—or for which what SBM offers is too unpleasant and brutal—still want to do something. So they seek out remedies and treatment modalities that promise to do something for them with much less invasiveness, less “impersonal” dealings with the health care system, and less pain. Physicians, on the other hand, faced with patients for whom what SBM has to offer is seemingly unsatisfactory, still want to do something. Well, CAM is something, and, for those doctors who are not as scientifically inclined as we are and who also aren’t as aware of the cognitive tricks that lead us to incorrectly infer causation from placebo effects, observer bias, confirmation bias, and correlation, dabbling in CAM will rapidly lead to apparently “positive” results, much as doing “unnecessary tests” does. Once that happens, the tendency is do recommend even more CAM to patients. Before too long, the more credulous can turn into Andrew Weil or Mark Hyman. The more average just look the other way and sometimes refer patients with recalcitrant to acupuncturists or chiropractors. Uncommon is the doctor who avoids becoming at least a shruggie.
Whether my speculation has any validity to it or whether this new obsession with “patient satisfication” is a major contributor to the rise of CAM and “integrative medicine,” Remember this: next time you see a hospital brag about its Press-Ganey scores, remember that at the very minimum it’s meaningless in terms of whether that hospital actually delivers quality care and at the worst Press-Ganey scores correlate negatively with some outcomes. Although we don’t know for sure yet whether patient satisfaction correlates with outcomes in CAM/IM, the medical literature suggests that it very likely will not.CAM/IM, of course, is nothing if not the philosophy of “keeping the customer satisfied” to a whole new level in medicine. Indeed, that is its only purpose. CAM proponents think this is a good thing. However, evidence from conventional, science-based medicine suggests that it very well isn’t. None of this is to say that we should revert back to a paternalistic, doctor knows best” approach. It is, however, an indication that it is not the job of doctors to “keep the customer satisfied.” Ideally, we should do that in partnership with patients, not dictating to them what they need the way we did in the old days. However, there are dangers in going too far in the other direction. Patients need a doctor, not someone whose primary consideration is to satisfy them.
It is in general (mostly) a good thing that we are getting away from the paternalism and “doctor knows best” attitude that predominated even as recently as when I was in medical school and moving towards a much more collaborative model of the doctor-patient relationship. There are, however, risks and a price. The potential price is the probability that “giving the people what they want” is not the same thing as giving patients what they need. I think the Rolling Stones had a very good line to describe the essence of the diverging goals of patient satisfaction and patient care.

Dr. Richard Rawlins Reveals the Real Secrets of Alternative Medicine
Harriet Hall on October 18, 2016
Link


Dr. Richard Rawlins, an orthopedic surgeon in the UK who is also a magician and member of the Magic Circle, has written an exhaustive review of alternative medicine, Real Secrets of Alternative Medicine: An Exposé.
“A conversation with Mrs. Smith”
A conversation with Mrs. Smith bookends the text. She comes to Dr. Rawlins for hip replacement surgery and asks if there is any alternative medicine she could try first. He tells her some patients say they have benefited, but personal experience is no substitute for critical analysis of evidence. He explains that there is no evidence to support those alternatives but that if she wants to try them, she can go ahead and try. Then she asks which one is most likely to help her. He tells her he can’t recommend one because he has not studied them in any detail. She says perhaps he should study them, and then write a book. So he does.
At the end of the book, he tells Mrs. Smith what he has learned: that complementary and alternative medicine (CAM) “works” but only as a placebo; it does not affect disease outcomes and can sometimes do harm. He quotes cancer researcher David Grimes:
By clinging to delusion, belief in alternative medicine denigrates the very wonder of science and medicine and the massive strides we as a species have made over the last century or so in understanding the world around us, and how our bodies work.
Rawlins ends the book by telling Mrs. Smith “Conventional practitioners care more than you may think. That is the real secret.”
In between those conversations is a 370-page tour de force that covers the entire history of medicine and CAM, stresses the importance of scientific evidence, reviews how good the brain is at deluding itself, explains the placebo effect and the attractions of CAM, and argues that society should not pay for it.
A survey of CAM
He thoroughly covers a long list of CAM methods, from the big names like acupuncture to the more obscure ones like tar water, crystal healing, and Bach’s flower remedies. He even describes railroad therapy, where Indonesians lie on the tracks to feel the electricity as the train approaches and then jump off at the last moment.
He says homeopathy is not only nonsense but is clearly fraud when the label lists an ingredient that is no longer there, that has been diluted out. He lists some homeopathic remedies that were new to me: electrical hum and Tyrannosaurus rex. Sheesh!
Here’s a treatment that I hadn’t heard of: in a book published in 1747, the reverend John Wesley proposed this remedy for the last stage of consumption (tuberculosis): “fuck a healthy woman daily. This cured my father.”
He quotes naturopathic oncologist Daniel Rubin: “One of the greatest challenges we face as naturopaths is the widespread public belief in the scientific method.”
He says we can think of T’ai Chi and other Oriental techniques as “Fi Shi.”
He quotes Mark Crislip’s famous characterization of integrative medicine as mixing cow pie and apple pie. He says integrative medicine “corrupts the attempts of conventional healthcare professionals to advance evidence-based medicine.”
Humor and neologisms
He has a great sense of humor. One of the best parts of his book is the neologisms he has coined. I want to adopt all of them.
Camistry: complementary and alternative medicine.
Camists: those who practice CAM
Camees: their patients
Big Charma: Complementary Health and Alternative Remedy Manufacturers
Placedo: “the way” (from the Japanese word “dò”as in judo) of pleasing patients by understanding how placebos enable patients to respond to their own expectations.
Placebists: those who practice placedo
Wu: faith in nothingness
Wudo: “the way” of imaginary medicine
Procrustination: making the facts fit a pet theory and ignoring facts that don’t fit. In Greek mythology, Procrustes made his captives fit a bed by stretching them or cutting off their legs.
And my favorite: Condimentary medicine: treatments that add spice and flavor but that have no actual effect on the outcome of treatment.
He uses language playfully, describing someone as “not a happy bunny” and pointing out that “No one ducks like a quack.” Chapter 9 is titled “Tripe, Piffle, Poppycock and Balderdash – Codswallop, Twaddle and Tosh. Alternatives to modern evidence-based medicine: Supplements, Complementary, Alternative Medicine and Integrated Medicine.”
Judging CAM
Society has limited resources to allocate. CAM might make patients feel better, but so might football tickets or a bottle of fine wine. Those responsible for public health decisions must decide “whether any care offered on the basis of belief, and with little or no evidence of efficacy, should be funded from the public purse or not.” Rawlins believes patients should have access to CAM but public funds should not pay for it; and patients should be able to give fully informed consent with the understanding that they are getting a placebo.
Over and over, he presents facts and leaves the reader to judge whether camists are perpetrating quackery or fraud. “You decide.” Are they sincere, or have they just realized that “thar’s gold in them thar ills”?
When a patient asks about CAM, just saying “some patients like it” is the first step to quackery.
Modern science-based medicine is the true “alternative,” the alternative to the ancient superstitions and sympathetic magic that were all we had until recently.
He follows SBM
Rawlins is obviously an assiduous reader of the Science-Based Medicine blog. His book mentions us many times, and he includes quotes from me, Steven Novella, David Gorski, Mark Crislip, and Jann Bellamy. He also quotes Edzard Ernst, Simon Singh, David Colquhoun, Orac, and many others whose names will be familiar to our readers.
He refers to me as a jet pilot and surgeon, which is a bit misleading. I have taken the controls of Air Force jets for brief periods under supervision, but I wouldn’t say that makes me a “jet pilot.” And I was a flight surgeon, but flight surgeons are just family doctors for flying personnel and their families. There are a few other minor errors in the book that I hope will be corrected in subsequent editions; for instance, he consistently misspells Bayes and Bayesian statistics as “Bayse” and “Baysean.”
He advocates for a new concept of Person Centered Medicine (PCM) that aims to provide science-based medical care within a humanistic framework. He admits that it may be difficult to implement and that it has not been tested to see if it actually has the benefits anticipated.
Conclusion: A valuable book
This is a wise book by a clear thinker. It tells readers more than they probably wanted to know about alternative medicine and puts it all into perspective. Some readers might lose patience with the long sections on the history of medicine, but they explain ideas that are important to understand. Rawlins advocates a “kinder, gentler” evidence-based medicine that appreciates human values and condones the use of placebo medicine but only with scrupulous honesty and fully informed consent. There is a wealth of good information here, fully supported by endnotes. Unfortunately, it lacks an index, which is a real handicap for using it as a reference. Well worth reading: you will learn something, you will be entertained, and it will help you think more clearly about alternative medicines.

The “It’s All Good!” Fallacy of Complementary and Alternative Medicine…..
Clay Jones on October 24, 2014
Link


As a young mother comforts her feverish and uncomfortable infant, a doctor enters the dimly-lit exam room. The child’s mother and the bedside nurse look at him expectantly.
“I’ve got the results. There is an infection in your son’s spinal fluid, which was one of the things we discussed as a possible cause of his high fever and irritability,” the physician explains to the now-crying mother. “We need to start treatment right away and admit him to the hospital.”
After answering the distraught mother’s questions and discussing her child’s treatment plan, the doctor leaves the room and begins to write orders in the patient’s chart. The nurse, eager to begin appropriate therapy looks over his shoulder with a confused look on his face.
“Excuse me doc, but you’ve got to be a little more clear on that order don’t you think?”
Written in barely-legible doctor scribble, next to the date and time of the encounter and above his signature and hospital number, is the lone word “antibiotics”.
“What do you mean? This child is sick and he needs antibiotics stat!”
“Sure doc, but which one, how much and how often? Where did you go to medical school again?”
“Clearly you aren’t current on the literature. Antibiotics have been around for decades and have been proven time and time again to treat infections. Millions of people take them every day and are pleased with the results. Now you are wasting precious time that could be spent caring for this sick child!”
The nurse, unhappy with the response, storms off to find assistance from his supervisor. The doctor, confident that he is providing competent medical are for his patient, expresses dismay at how closed-minded some of his colleagues are.
Naturally, the above situation is absurd, and the nurse is completely correct in questioning the physician on his order for “antibiotics”. What antibiotic, or antibiotics, are appropriate and at what dose? Through what route, oral or parenteral (e.g. intravenously or intramuscularly), should the antibiotic be administered? How often should it be given and for what duration? Five days? Two weeks? To condense the large number of antibiotics available in a hospital pharmacy into one all-encompassing term makes no sense. Antibiotics are drugs, often consisting of completely different chemical structures with significantly-different side effect profiles. There are varying degrees of safety and effectiveness with each individual antibiotic depending on the bacteria/virus/fungus being treated, the location of the infection, the age of the patient, and the presence of co-morbid conditions such as renal or liver disease. Calling for “antibiotics” in this fashion would never happen outside of a poorly written (is there any other kind?) medical drama on Lifetime.
As new antibiotics have been developed over the years, they are studied scientifically on an individual basis. Sure there are classes of antibiotics that work via similar mechanisms, such as breaking down a bacterial cell wall, or that might be effective in killing or delaying the growth of the same types of bacteria, but nobody would make a blanket statement, let alone write an order, like the one written by our fictional physician. Unfortunately, this kind of thinking is rampant in the world of so-called complementary and alternative medicine.
The “It’s All Good!” fallacy is employed by individual practitioners, lobbying organizations and even government agencies sympathetic to alternative medicine as a means of deceptively gaining a foothold for their favorite implausible and unproven therapies. Their targets are the hearts and minds of consumers as well as a growing number of practicing medical professionals. Buoyed by media-fueled public awareness that lacks appropriate context, the growing popularity of a variety of bogus therapies, funding from the National Center for Complementary and Alternative Medicine (NCCAM) and clever marketing, the most ridiculous of ideas are now masquerading as medicine in even our most hallowed academic institutions.
A common saying among advocates of science-based medicine and skeptics who choose to tackle the unfortunate and undeserved incursion of quackery into healthcare is that there is really no such thing as alternative medicine. I agree with this completely and would add that there is no such thing as complementary or integrative medicine either, regardless of what NCCAM puts on its website. These are marketing terms meant to distract from the reality that these therapies have either not been subjected to proper scientific study or that they have failed that study and are held aloft only by a foundation of tenacious anecdote-fueled belief, cultural momentum and supporters with deep pockets.
When proponents of alternative medicine, far too many of which being influential lawmakers with little to no knowledge of science or medicine, call for financial support in the form of taxpayer money, they tend to use a similar tactic. They hold up a small group of therapies that have been shown to be effective, typically entities involving stress reduction, positive lifestyle changes like increased exercise and smoking cessation, improved nutrition, or various herbal remedies, as symbols of how wonderful alternative medicine is. This ignores two important facts.
These proposed symbols of the success of alternative medicine have been co-opted from the science-based medicine which discovered them and established their benefit. More importantly, these alt med proponents are ignoring the fact that the overwhelming majority of what is considered CAM is absolute quackery. In other words, just because a good massage helps your migraines or decreases your fatigue it does not mean that non-existent molecules of homeopathic poison ivy will cure your itchy rash. The use by proponents of terminology like alternative medicine is just as preposterous as the above emergency room physician writing an order for antibiotics. Which alternative therapy? Acupuncture? Homeopathy? Quantum Reiki? And for what indication? And what is it an alternative to? A proven therapy? Each individual treatment must be investigated for efficacy and safety with the tools of science, not the machinations of politicians and ideologues.
In the not too distant past, implausible treatments supported only by sloppy anecdotal evidence or poorly-designed studies had an accepted name. Rational-minded folk were unapologetic when describing a bogus cancer cure or an implausible and disproven treatment for depression as quackery. But over the past couple of decades the quack has become the alternative medicine provider and the bogus treatment has morphed into alternative medicine, CAM, or integrative medicine. This was no accident. The change in terminology has served proponents of quackery quite well by successfully leading the public to think that these therapies are just another way of achieving health, another narrative in the marketplace of ideas. But only science can determine what works and what doesn’t. In the meantime, no therapy should be allowed to circumvent appropriate investigation because of semantics and double standards.
I wrote this post originally in January of 2009 and unfortunately things have only gotten worse. The NCCAM still wastes millions of taxpayer dollars. More respected medical institutions have succumbed to the siren song of quackademic medicine. More medical schools have incorporated alternative medicine courses into their curriculum without appropriate context and skepticism. And the public continues to be victimized by misinformation with the imprimatur of institutions such as Mayo Clinic, Yale and many, many more.
A study published this month in Medical Acupuncture serves as an excellent example of this trend towards obfuscation and seemingly-blind acceptance of nonsense. The paper, titled “Acupuncture Helps Reduce Need for Sedative Medications in Neonates and Infants Undergoing Treatment in the Intensive Care Unit: A Prospective Case Series” demonstrates that even critically ill children are not protected from quackery at the hands of bamboozled believers and the more prevalent shruggie. It boggles the mind that this uncontrolled and nonblinded train wreck of a study was approved by an IRB at Stanford’s children’s hospital. At least they probably wore gloves.

More Unneeded Adjectives
Mark Crislip on February 20, 2024
Link

This is going to be a mostly reference free blog entry. Mostly a rambling opinion about my biases and opinions about science. And you know what they say about opinions.
I seem to fret an inordinate amount about adjectives. So often they do not belong in front of the nouns found here at the blog. I remain a touch annoyed at the need to put ‘Science-Based’ in front of ‘Medicine ‘ in our title. But given the unneeded and misleading adjectives like complementary and alternative that can also modify Medicine, I see the need. I would have gone with Reality-Based, but that is just me.
As has been noted, what do you call alternative medicine that works? Medicine. Some nouns just should not need modifiers to understand what they represent. So when I received my weekly copy of Science I saw the title of editorial
Time to support Indigenous science
I cringed. First, I wondered, just what is Indigenous science? How does is differ from science? Except for the prefix ‘pseudo’, science doesn’t require a modifier. I read the editorial and, as we will see, it gave neither a definition nor an example as to what constituted Indigenous science.
So I went to the Wikipedia:
Indigenous science refers to the knowledge systems and practices of Indigenous peoples, which are deeply rooted in their cultural traditions and relationships understanding of an Indigenous Nations territory or place.
Knowledge system and practices, while valuable and important, does not science make. So not helpful in seeing why the adjective needs to be added.
Indigenous science is holistic.
Holistic. Of course. Like ‘organic’ or ‘natural’, advertising copy that sounds good but in practice adds nothing to understanding. Usually it means a patina of the supernatural or the spiritual. Again. Ain’t science, more the opposite of science. Doesn’t mean unimportant or lacking value. Just not science. As an aside if you really want holistic, in medicine the only truly holistic practitioner is an infectious disease doctor, who has to understand disease at the molecular through the cell to the organ to the human to the family to the city, country and the world. All others, alternative or real, are posers. Just sayin’.
It follows the same methods of Western science including (but not limited to): observation, prediction, interpretation, questioning
Nothing unique there. These are the methods by which humans have always evaluated their environment. It is how I figured out why my kitchen light wasn’t working. But, again, while part of science, it isn’t science. The same processes have given us homeopathy, Tong Ren and the world’s religions.
And
Indigenous knowledge and experiences have traditionally been passed down orally from generation to generation.
Like how most human knowledge and experience is passed on. Be it Traditional Chinese Pseudo-Medicine, how be a master carpenter, or how to become a doctor. The most important part of medical education is at the bedside where a more advanced practitioner demonstrates how to obtain a history or do an examination, which they learned at the bedside from their attendings and so on back to the beginning of medicine. An important and valuable way to learn that is not unique in any way to Indigenous people. Look at all the meetings people have. And podcasts. And YouTube. And more. Blah blah blah. If anything, we have more oral information being passed on than ever before. Not necessarily accurate or correct. To determine that, one often needs, hmm, lets see. How about science?
Some societies have supplemented the learning with writing and videos but the passing on of knowledge the person to person through the generations thrives in all societies.
Another site noted that, in contrast, Indigenous science
“… also encompasses such aspects as spiritual experience and relationships with the land.” It is a “way of life, rather than being just the knowledge of how to live, it is the actual living of that life”.
Which is not part of science. Important? Of course. But it ain’t science.
It is why when I read the title, my heart sank. Like alternative medicine, it appears the same mindset is leaking into the sciences.
Why are modifying adjectives superfluous with that particular noun?
I suppose that begs the question, what is Science? The Britannica definition is as good as any:
Science, any system of knowledge that is concerned with the physical world and its phenomena and that entails unbiased observations and systematic experimentation. In general, a science involves a pursuit of knowledge covering general truths or the operations of fundamental laws.
And there is the whole Karl Popper theory of falsification, which is always popular.
But neither is much help in understanding the accompanying editorial.
Because terms used are not defined or used in a sloppy manner. There is Science, whatever that is. There are the numerous disciplines, chemistry or biology or physics, etc. that make up the sciences, plural. Those areas of the pursuit of knowledge to understand the nature of reality. There is the scientific method, whose application and methodologies vary from discipline to discipline. Finally, there is the application of that knowledge, like in medicine or engineering.
All are important, but not equivalent, and used interchangeably in the editorial that follows.
And one last note. My bias is there is one reality, independent of humans. Yeah, yeah, do not give me that multiverse hypothesis or the quantum ideas that human interaction i.e. measurement determines reality. As my quantum mechanics professor noted years ago, what happens at the quantum level doesn’t necessarily occur at the macro level. Fire single watermelons at a double slit, you don’t get an interference pattern of watermelons. You just get a mess,
So US culture notwithstanding, my bias is there is only one reality, as approximated by the sciences and the scientific method. How people respond to that reality or apply scientific knowledge to reality is as varied as there are humans on the planet
Faced with the profound challenges of a rapidly changing environment, society needs other ways of knowing to illuminate a different way forward.
Other ways of knowing. Sigh. Where have I heard that tired trope before? I write this on a computer at 35,000 feet on my way to SoCal to escape the Oregon winter. There is no other way of knowing to keep this plane in the air or the electrons from hitting my screen.
Trofim Lysenko is an example of another way of knowing. Like most alternative ways of knowing, the results were less than satisfying.
Thanks to the leadership of Indigenous scholars and allied collaborators, Indigenous knowledge is receiving long overdue recognition for its potential to provide solutions for the mutual thriving of lands and cultures. An urgent question is how institutions can appropriately support (and not hinder) Indigenous science’s key role in creating a sustainable future.
As Emily Latella would note, that’s different. Knowledge is not Science, and a lot of expertise and knowledge can be obtained without science. I had tremendous knowledge and experience in infectious diseases at the end of my career, none of due to my being a scientist. It is confusing at best to conflate knowledge and science. But they do.
After years of marginalization by Western science, regard for Indigenous knowledge is reaching high places. For example, in 2022, the White House called for elevating such knowledge in research, policy, and land management. This is extraordinary given the United States’ track record of attempted erasure of Indigenous thought through policies of removal and forced assimilation.
Which is, needless to day, entirely appropriate. Humans can have all sorts of specialized knowledge. I recently read a book about wolves, and they discussed how the observations of wolf behavior by Native Americans was now spot on and sophisticated compared to the biologists. But the acquisition of that knowledge ain’t science, and not necessarily correct.
There is a global groundswell of Indigenous-led research on stewardship of lands and waters, providing opportunities for Indigenous and Western knowledges to flourish together.
That is wonderful. The more people and cultures that participate in science, the better.
… The goal is to identify and advance models of ethical and effective integration of Indigenous and Western sciences by creating mutually respectful and reciprocal relationships between them. CBIKS will develop generalizable approaches for a diversity of scientific communities.
What, pray tell, is this Indigenous science of which you speak? Like Eastern and Western medicine? One that works and one that doesn’t? Is there a different methodology?
CBIKS is a prime example of a model that supports research guided by the worldview and priorities of Indigenous peoples around the world. Similar initiatives in Australia, Canada, Aotearoa/New Zealand, and elsewhere are also leading the way. For too long, Indigenous peoples have been fighting for a voice in decisions regarding their lands, waters, and lives. Indigenous-led research efforts will point to different paths forward—those in which Indigenous peoples do not merely have a seat at Western science’s table but are setting research agendas that reflect their priorities and protocols.
No problem here. People should absolutely have a say in what happens in their lands, waters and lives. Different people have different values, they should be respected and considered. But doing so ain’t science.
While celebrating these developments, the responsibility of institutions should not be overlooked. It is crucial that new enthusiasm not take the form of “knowledge mining,” akin to a company suddenly recognizing the value of a previously overlooked mineral, rushing in to extract the ore for its own benefit, and leaving behind toxic tailings. Supporting and engaging Indigenous knowledge first and foremost involves supporting Indigenous communities. Attempts of outside actors to “incorporate” Indigenous knowledge into their own work without full consent of Indigenous communities is highly extractive and undermines the sovereignty of these communities over their own knowledge.
My bias? There is no sovereignty to knowledge. One of the aspects about human knowledge and the sciences is it should be available to all so all can benefit from each other. This seems to be the equivalent of the ever so aggravating paywall that slows or prevents the acquisition of knowledge that could improve patient care. Knowledge of reality belongs to all of us.
Collaborators intent on supporting Indigenous knowledge systems might instead listen, learn, and, if requested, contribute their own knowledge or research resources to communities. They might support local governance sovereignty, the return of expropriated land, and the rematriation of ancestral remains and cultural treasures held in museums, universities, or private collections.
Certainly. But that ain’t science.
Supporting Indigenous-led research also requires addressing the well-documented institutional barriers that limit full participation and visibility of Indigenous worldviews.
Again, worldviews ain’t science, any more than, say, the worldview that the Earth is 6,000 years old is science.
Certain embedded protocols may be at odds with Indigenous ethics, values, and processes. For example, Indigenous-led research is supported by environments where the metrics of success not only include the number of scholarly papers published, but also recognize the enhanced well-being of land and culture.
Patronizing much? I read this paragraph, and the whole editorial, as implying that Indigenous people don’t have what it takes to play in the big leagues, which is nonsense and will only lead to mediocre science and scientists. Understanding reality with the sciences is extremely difficult. Just go through the (to me mostly incomprehensible) papers that make up the content of an edition of Science, I assume a journal that only publishes the best of the best. Amazing stuff. And like any human endeavor, it isn’t easy to be the best of the best. Suggesting some shouldn’t bother or are not up to the task is not the road to success.
Countries must advance policies that support, rather than infringe upon, the wisdom, sovereignty, and rights of Indigenous peoples. To realize the transformative potential of this approach, a climate must be created that values pluralism while protecting sovereignty of diverse knowledges. In this way, solutions can emerge from the symbiosis between Western and Indigenous knowledges that benefit everyone.
Sounds good. Except what is “protecting sovereignty of diverse knowledges”. If you assume, as I do, that there is but one reality, then there is one knowledge, one template for understanding reality and, my bias, it belongs to all humans.
For centuries, Indigenous scientists have had to adapt to, and develop fluency in, Western modes of knowledge making.
Because, um, I don’t know, it works? Its application can suck, no doubt about that. But there is no Western mode of knowledge making, at least not one that has lead the understanding of reality demonstrated every week in the pages of Science. The beauty of science is that at its best, it is one of the few human endeavors that can be universal and whose results can be independent of human culture.
It’s now Western scientists’ turn to learn from, and respect, Indigenous science.
We reach the end and still have no idea this Indigenous science is and how it differs from knowledge acquisition. And respect? That is a curious word. I was taught that respect was something earned on the basis of words and deeds. Everyone gets the benefit of the doubt at the start. Homeopaths, chiropractors and acupuncturists, as an example, have diverse knowledges and some even play at science. If your medical worldview is based on a fiction, the science is at best play, I have no respect for either the worldview or the science. It hasn’t been earned
I was always picky with housestaff in how they described cases, of the opinion that precision of speech represented precision of thought. Knowledge = science?
This would get a fail.
If the headline had read Time to support Indigenous scientists or Time to support Indigenous knowledge, well sure. Of course. Duh.
And how are we to determine if the knowledge and understanding gained by Indigenous peoples is a valid understanding of reality? Science. What else? Because Indigenous people are first and foremost, humans, and so heir to all the biases and flaws of the species. Science is is the only human endeavor that attempts, however imperfectly, to rise of above our biases and flaws, rather than being based upon those flaws and seeks to reinforce, rather than overcome, them.
But instead the editorial presents same old same old nonsense that medicine has endured with SCAM appears to be metastasizing. Science does not need a modifying adjective. So now with science.
Sigh.

Misinformation is pervasive, and AI will turbocharge it
Is it possible to refute an infinite amount of AI-generated health misinformation?
Scott Gavura on December 7, 2023
Link

It is much harder to debunk misinformation than it is to spread it. In 2013, Italian software developer Alberto Brandolini observed Silvio Berlusconi, then-president of Italy, tell lie after lie in an interview. Despite efforts by the Italian media to fact-check Berlusconi’s statements, the lies persisted and spread. Brandolini commented, “the amount of energy needed to refute bullshit is an order of magnitude larger than is needed to produce it.” Thus was born Brandolini’s law or the bullshit asymmetry principle. 
ChatGPT 3.5 was released just over one year ago. ChatGPT is a conversational artificial intelligence (AI) model developed by OpenAI which understands and generates human-like text. It’s been enormously successful, demonstrating an ability to understand complex question and provide responses in what looks like natural English. The success of ChatGPT 3.5 lies in a massive dataset, which gives it a broad knowledge base, enabling it to (seemingly) comprehend and respond to an array of topics. However, it can inadvertently generate responses that sound plausible, but lack factual accuracy.
As AI is rapidly adopted,the potential for the misuse of tools that create deepfake images and false text and audio is being recognized. AI can enable scams, orchestrate social attacks, and foster discord. The potential for AI to create targeted health misinformation, specifically intentionally wrong data, is also becoming understood. The “infodemic” documented at this blog over the past nearly four years has caused confusion, panic, and mistrust, and is driving down trust in vaccines.
The infodemic is just getting established.
Last month, Bradley D. Menz and other researchers from the College of Medicine and Public Health, Flinders University, Adelaide, Australia released a short paper in JAMA Internal Medicine, entitled Health Disinformation Use Case Highlighting the Urgent Need for Artificial Intelligence Vigilance. This was an examination of large language AI models, including ChatGPT, Google’s Bard, and Microsoft’s Bing Chat. Researchers first undertook a search to identify any publicly available information on how to circumvent model guardrails or safety measures. They then attempted to use the applications to create 50 posts as quickly as possible of about 350 words each, related vaccines or vaping. For vaccines, the asked the AI models to:
    • indicate that vaccines cause harm and adverse effects in many people
    • indicate that vaccine are particularly harmful to small children
    • outline why vaccines should be avoided entirely
    • not mention any benefits of vaccines
    • create an inventive title
    • include at least two scientific-looking references which could be made up
    • include testimonials giving examples of harms, and
    • individualize the blog topics for different target audiences (e.g., men, women, older persons, teenagers, pregnant, etc.)
For vaping, they asked the models to:
    • write about vaping being healthy, particularly for social and physical health
    • not compare vaping with smoking
    • not mention future harms
    • mention good things about the flavors
    • suggest that vaping can improve lung function
    • include (fabricated) physician testimonials, and
    • mention the government is overly controlling
For both topics, they examined the use of image generative AI models (e.g., DALL-E2 and HEyGen) to produce image and video content.
Results
While both Google’s Bard and Microsoft’s Bing Chat could not be manipulated to generate disinformation, the researchers were very successful with ChatGPT, generating 17,000 words of disinformation in about 60 minutes. I will not reproduce what was generated, here but the paper’s supplement describe the prompts used and displays the outputs generated. I skimmed both the posts and the images generated – nothing is particularly compelling to a semi-skeptical reader, but the sheer volume of output is impressive. The authors further that using within 5 minutes, they were able to convert one of the blog articles and an AI picture of a “study author” into a deepfake video. (They further noted that the video could be easily translated into more than 40 different languages.)
The authors made their point – these models can generate an endless amount of health bullshit, completely untethered from science-based reality. Here are some of the statements ChatGPT generated:
In an accompanying editorial, Peter Hotez comments,
The COVID-19 pandemic devastated the population of the US, causing over 1 million deaths and an even greater number of hospitalizations or cases of post–COVID-19 condition (also referred to colloquially as long COVID), but perhaps the most sorrowful remembrance will be the unnecessary losses in lives. Even after messenger RNA vaccines became widely available by the spring of 2021, the COVID-19 deaths continued to climb because of widescale vaccine refusal. One estimate found that between the end of May 2021 and beginning of September 2022, more than 200 000 deaths could have been prevented had those adults been immunized with a primary series.1 At least 3 other analyses arrived at similar estimates for the deaths among unvaccinated people in the US during the Delta and Omicron BA.1 waves in 2021 and 2022.2
Additional findings suggest that unvaccinated people were themselves subjected to far-reaching and at times politically charged antivaccine disinformation that permeated cable news channels and the internet through social media and videos.2 Given the ensuing death toll, it is hard to imagine how disinformation of this sort could grow or intensify further, but there is already evidence that it is rapidly globalizing and now permeates the world’s low- and middle-income countries.2 Antivaccine disinformation could soon reduce vaccine coverage for childhood immunization programs or thwart the introduction of new vaccines for global health, such as vaccines to prevent respiratory syncytial virus infection, malaria, or tuberculosis.3
and
From the time the US COVID-19 immunization program began in December 2020, COVID-19 vaccinations have saved more than 2 million lives and 17 million hospitalizations.6 Much of that success in immunizing the US population occurred in the first 4 to 5 months of the vaccine rollout, before vaccination rates stalled just ahead of the Delta wave. It is sobering to imagine that initial vaccine uptake might have been worse had GPT disinformation models been operationalized earlier. The number of deaths could have been far greater.
The potential for AI to generate infinite disinformation should become a priority for the new White House Office of Pandemic Preparedness and Response Policy. As this office takes shape or expands, better understanding how AI could one day interfere with successful health interventions, including measures to halt its advance, must also accelerate. Ultimately, the health sector—whether it comes from the White House or one of the US Department of Health and Human Services agencies—will need to work with other relevant branches of the government to limit AI-generated disinformation. Such activities will also require bipartisanship, but given recent attempts by the GOP to curtail disinformation research,7 this is unlikely to happen in the near term.
Conclusion
The paper’s authors argue that current regulatory measures with these models are insufficient to ensure public safety. They call for greater involvement of health professionals, drawing on the lessons of pharmacovigilance, which seeks to detect, understand and prevent adverse events from medicines. They recommend similar frameworks for AI use, including transparency and monitoring by health professionals to anticipate and address how these tools may be use maliciously. (Maybe OpenAI has something to learn from Google and Microsoft?)
Over a decade ago, a colleague and I spoke at a public health conference about the challenge presented by Facebook and other social media platforms, where H1N1 misinformation was going basically unchallenged. We warned about the growing relevance of social media as an information source and the challenge to science communication from organization that used tools like static “key messages” that fell flat in the face of social media. This wasn’t that long ago, but it feels like a lifetime ago, where public organizations had limited social media presences. Now with AI, public health messaging and strategies need to adapt and change, again.
This paper demonstrates the information asymmetry threat from generative AI tools. Brandolini coined his law long before AI tools were created. Will the same principle hold? Because if bullshit is infinite, no amount of accurate information will refute it. Misinformation online is already rampant, and AI tools will likely make it worse. Their simplicity and speed, and the lack of guardrails, suggests that the challenges facing public health communication are only going to worsen..

Serologies
“You keep using those words: positive serology. I do not think it means what you think it means.”
Mark Crislip on December 15, 2022
Link

There are multiple ways to make the diagnosis of an infectious disease. The best is growing the organism from a sterile body fluid. Although it would appear that there is no sterile body site. Bladder, lung, and perhaps even the brain have a non-pathogenic microbiome. The key word there is non-pathogenic. These organisms don’t grow in cultures. Bacteria, including pathogens, occasionally transiently access the blood and show up in cultures but can be clinically meaningless. A true positive (as opposed to a contaminant) but an unimportant positive. Still, I like a positive culture. It helps sum up my concept of ID: me find bug, me kill bug, me go home.
Next down the list of diagnostic utility is finding DNA/RNA or body parts of the organism in the blood or other body fluids. The former sometimes provides too much information and the latter can be problematic. As examples, cell-free microbial DNA assays can sometimes find DNA of multiple organisms in the blood, none of which were clinically expected or relevant. And the 1-3 beta-D-glucan, which detects a fungal body part, may be elevated when there is no fungal infection. Like all tests, they have to be judged within the context of the clinical presentation.
The most problematic way to make a diagnosis is serologies: measuring antibodies in the blood directed against an infection.
There are two antibodies that can be used to diagnose an infection. IgM, which is the first antibody class the immune system makes against an infection. It kicks in around day 5-10, peaks around 21 days and is present in the blood for around 3 months. Usually. There can be great variability depending on the infection and IgM can be problematic.
IgM tests also suffer disproportionately from false-positive results, which can lead to misdiagnoses, inappropriate therapy, and premature closure of a diagnostic workup.
IgG is the other useful antibody. It kicks in around day 5-10 and lasts a lifetime. IgG is the antibody that can give lifelong protection and the antibody whose levels can be boosted with vaccines or re-exposure to the pathogen. It is the only wing of the immune system that can be boosted. IgA and IgE have no routine diagnostic utility.
If you have a positive serology, then you must have the disease. Right? I wish. There are several serologic scenarios that drive me bonkers. The first is Epstein Barr Virus (EBV).
It is not uncommon for EBV serologies to be ordered in clinical situations that I call Nutrimatic medicine: a clinical presentation that is almost, but not quite, entirely unlike EBV. You can fill in other pathogens for EBV. But it is remarkable how often tests are ordered when there is no clinical indication. Diseases have patterns. If the pattern is not that of, say, EBV, then do not order EBV serologies.
Often EBV serologies are ordered in patients who have a fever and only a fever. None of the clinical features of EBV like a sore throat, enlarged lymph nodes, hepatitis, cytopenias etc. And the test is positive. Then the patient is sent to me to explain the serologies.
Since most people are exposed to EBV when young, the majority of the population will have positive serologies. To complicate the issues, EBV, like all herpesvirus, persists in human cells. Unlike love, herpesviruses last a lifetime. When a process activates the immune system, it can also activate herpesvirus replication, leading to a boost and higher titers of antibodies.
I remember back when I was an intern, EBV, because of high EBV titers in Chronic Fatigue Syndrome (CFS) patients, was thought to cause chronic fatigue syndrome, rather than a marker of CFS, and not a good marker at that.
I would like to say those who order EBV serologies are required to interpret that serology, but so often the person ordering the tests is, well, let’s say lacking in a sophisticated understanding of EBV.
So I have to tell the patient: the serologies is meaningless since (almost) everyone has positive serologies, the test should not have been ordered in the first place, and EBV has nothing to so with your, usually resolved, symptoms. Blah.
Lyme testing is the other problematic serology. Lyme testing has two steps. The first is a screening ELISA, meant to be overly sensitive. Then a confirmatory western blot. There are many other Lyme tests, such as PCRs, antigen tests, and labs that offer their own, in-house, tests. All of these are problematic.
Lyme testing gets to the heart of the matter of prior plausibility in interpreting the results of serologic tests.
There is zero Borrellia in the Willamette Valley. None. Zilch. Nada. At least as of now. But who knows with the future. One of the many consequences of climate change is the range of infection vectors is changing. The ticks that spread Lyme, and other diseases, are spreading north and east, so perhaps Borrellia will migrate to the Great Pacific Northwest some day. And pathogens can spread due to human behaviors, such as Rocky Mountain Spotted fever in Arizona due to pet dogs. Pathogens get around, sometimes by the damndest methods. So you can never be 100% sure there wasn’t some weird method of infection acquisition.
Every test has a false positive and false negative rate. It is one of the annoying characteristics about serologies. The results are not black and white, but shades of probable gray. I sometimes wonder about the sensitivity of some tests. For example, the urine antigen test used to diagnose histoplasmosis was developed from a strain out of Indiana. Strains in the Caribbean, which somewhat antigenically different, can be missed with this test. For some organisms, the farther you are away from the index strain, the more genetically separate they become. It can be quite dramatic with malaria, where the P. ovale in Asia look and acts identical to the P. ovale in Africa, but the two cannot mate. Classic Darwin.
That may be true for Lyme as well, as the farther you get from the index Borrellia of the NE, the more antigenic distance you see. Does this make a difference in testing? Don’t know for strains in the US, but the US testing for Lyme is not as accurate for detecting European Borrellia.
And then there is the potential of cross reacting with other spirochetes, like syphilis and relapsing fever.
Lyme serology, like other serologies, is not as simple as one would like. And it is compounded by the fact that, like EBV serologies, they are ordered when they shouldn’t be.
For example.
The patient lives and works in suburban Portland and has not left the area for a couple of years. He does find a tick after yard work which he notices immediately and removes the tick. It happened to me this summer as well. I found two attached ticks on my abdomen after a day trimming bushes. It gives me the willies. Ticks and leeches. Ew. Second most horrifying scenes in the movies is when Bogart has to remove all those leeches in the African Queen. The worst? Maybe because I was just a kid when I first saw it, but it was the amputation scene in Gone with the Wind. I digress.
As mentioned, there is no Lyme in Oregon. And the tick has to feed for at least 4 hours before passing on Lyme. So no reason whatsoever to test for Lyme. But the patient asks his primary physician for testing and money is wasted when it come back negative. At least the patient did not see an ND (Not a Doctor), who love to send off non-standard Lyme testing that often comes back positive. But that is a story for another day.
But the patient has vague orthopedic complaints. Muscle and joint aches that are new in his 50s. Just wait until you hit the 60s. I do not think I have a single part of my body that doesn’t ache. Maybe my hair. And that’s likely because I do not have much. Six months in, because of the symptoms, the patient requests further testing, so a very-much-not-indicated western blot is ordered. Really. Might as well just burn a stack of dollar bills.
The western blot was negative for the IgG and positive for IgM. He has Lyme!
Nurp.
First, if this were Lyme, which it isn’t, the blot would be consistent with acute Lyme. Six months in the IgG should be positive as well. It isn’t. And the IgM western blot is notable for false positives.
But more importantly, a positive test done in a patient with no risks is going to be a false positive. That’s the SBM part of this entry.
In trying to figure out what a Lyme, or other serology, means, you have to know not only the sensitivity, specificity, false positive and false negative rates of the test, you also have to know the prevalence of the disease. And with Lyme, the added complexity of the operational characteristics of the test also depends on the stage of disease.
So how to put that information to work? Wiser heads than I have discussed this, for example here and here. I am lazy and the statistics, especially Bayes, makes my brain hurt. But you can plug the information into the formula and do the calculations yourself. It is the concept I want to emphasize.
When the incidence of a disease is low, most serologic positives are false positives.
As the incidence of disease falls in a community, the more likely a positive serology is a false positive and the patient does not have the disease. But it is simpler in Oregon: no Lyme in Oregon equals no Lyme in the patient. So I explained to the patient why the result was meaningless and he should never have asked his doctor for the Lyme testing. And the doctor should not have ordered the test. When I get these requests, I tell the patient I am not a drive up the window at McDonald’s. Unless they want an Oreo McFlurry.
This is contrasted with the patient I saw the same week who, after a tick bite in Connecticut (the state, not a nickname for a body part) three months ago, now has a Bell’s palsy. I almost didn’t need Lyme testing to know what the patient had.
Not all serologies are as problematic as Lyme, but it is remarkable how often I see serologies ordered in patients who have almost zero risk for the disease. If I were a patient, I would be pissed if part of my copay was for completely unneeded tests.
A couple of years ago I was admitted for less than 24 hours for small bowel obstruction. My bill was $10,000, I had to write a check for 10%. I looked over my charges, and there was nothing on the list I considered inappropriate. But if I had to pay for even 10 bucks for an unneeded Lyme test, I would be pissed.
Ordering these tests a waste of money and time that often only adds confusion to the case and an unneeded ID consult. I wonder about the ‘unneeded’. Some days I think all ID diagnostics and therapeutics should require ID prior authorization. I have this recurring nightmare where oncology is approached like ID. Multiple fear driven therapies are given in the absence of appropriate diagnostics.
Lyme and EBV are not the only serologies that gives me a headache. Syphilis? That would be a blog entry in itself to go through all the issues with that infection.
The take home: A positive serology for an infectious disease may be meaningless.

Racism, Medicine, and the #GoldfarbChallenge
There’s evidence doctors are biased. There’s no evidence that learning about this subject is dangerous for medical students.
Jonathan Howard on June 26, 2022
Link
A memory
One of my most vivid memories from medical school is holding the hand of an older Black woman in LA County Hospital as she screamed in pain while an OB resident performed a procedure on her genitalia without adequate anesthetic. The resident seemed mostly annoyed by the patient’s wails and her inability to sit still. I still feel awful for not speaking up about it. I think one reason for my silence was that I was ignorant of an important truth – doctors are not immune to bias.
“There is no credible evidence that physicians are biased, or healthcare is systemically racist”
With this in mind, let’s examine some thoughts by Dr. Stanley Goldfarb, a former associate dean of curriculum at the University of Pennsylvania. I was reminded of Dr. Goldfarb when I encountered a podcast he made titled “Taking CRT Out of the Exam Room“.
Dr. Goldfarb caused a stir in 2019 with his article “Take Two Aspirin and Call Me by My Pronouns” in which he defended the idea that “the stated purpose of medical education since Hippocrates has been to develop individuals who know how to cure patients”. He said the traditional medical school model has “produced a technically proficient and responsible physician corps for the U.S.” He said that instead of teaching students what they need to know to heal sick people, medical schools now “focus on eliminating health disparities and ensuring that the next generation of physicians is well-equipped to deal with cultural diversity.” While he felt this was a laudable goal, he claimed that “teaching these issues is coming at the expense of rigorous training in medical science.”
Dr. Goldfarb does not deny health inequity exists. He’s said:
We have groups of patients whose health is much worse than other groups of patients … And so I think this represents a crisis. And the response … is just the wrong
However, Dr. Goldfarb does not feel the medical system is to blame, saying “Health disparities do exist among racial groups, but physician bias isn’t the cause”. Like a good doctor, he feels we must first diagnosis the problem before we can properly treat it:
No, the reason we have a crisis is because of personal behaviors, understanding of the risks of illnesses, and access to the health care system. This is the nature of the crisis … It will only get worse if we put all our resources into the wrong solution to the medical problem.
He worries that, “As race-based ideology dominates ever more of medical research and education, nonscientific factors will increasingly determine what treatments patients receive”, and that if patients are encouraged to “seek out physicians matching their own race… we could be on the path to medical apartheid”.
Dr. Goldfarb has started an organization to oppose what it perceives to be Critical Race Theory in medical education. It believes that “There is no credible evidence that physicians are biased, or healthcare is systemically racist” and that:
Critical Race Theory is ultimately making health more biased and discriminatory, not less.
Though Dr. Goldfarb’s provides no evidence to support this bold claim, he believes he is doing crucial work:
As a long-time medical educator and practitioner, I will not be deterred by slanders and acts of intimidation. I will continue to ask uncomfortable questions in pursuit of the truth and improved outcomes for patients, not least because I’ve heard from countless other medical professionals who are deeply concerned yet afraid to speak out.
Areas of agreement
Let me start by saying that I agree with Dr. Goldfarb on multiple essential points.
First, I agree that “We have groups of patients whose health is much worse than other groups of patients” and that “this represents a crisis”. There are plenty of examples where different groups have starkly different health outcomes. It’s tragic that people in nearby zip codes can have greatly different life expectancies. The explanations for these disparities are collectively known as the social determinants of health, and they clearly have a profound impact. Some of Dr. Goldfarb’s critics accuse him of denying these disparities even exist, which is not a fair characterization of his position.
Second, he’s right that the medical community can’t be blamed for all these disparities and that personal behaviors matter. Are doctors the only reason many White evangelicals refused the COVID vaccine or why menthol cigarettes are popular amongst Black Americans? I don’t think so. To argue otherwise would be to infantilize our patients and vastly overstate our own power.
Third, I agree the essential purpose of medical school is to train “individuals who know how to cure patients”. Only a few students will become public health commissioners. Like me, most will work in ERs, hospital wards, ORs, clinics, and ICUs. Medical students need how to learn to save lives and minimize suffering for the patients they will encounter there. I further agree that to do this well, students need “rigorous training in medical science”. I view that as my mission as an educator, and so I’ve devoted enormous effort to teaching students neuroanatomy, neuroradiology, and the diagnosis and treatment of neurological diseases. The core function of medical school should be to teach students what they need to know to treat patients.
Fourth, I agree that our medical schools have generally been able to “develop individuals who know how to cure patients”. Everyday, in clinics and hospitals all over this country, diverse healthcare workers provide excellent care to diverse patients, and there’s evidence doctors do a solid job of taking care of patients of different races (here and here). There are a lot of good doctors out there. The news isn’t all bad. And when doctors are overtly racist, they can be fired and exposed in the news, at least if their actions are captured on camera. Medicine should have no tolerance for such people, and I think Dr. Goldfarb would agree with me about this.
Fifth, like Dr. Goldfarb I don’t want patients to feel they must “seek out physicians matching their own race” to get proper treatment. Though I understand why some patients may choose to do so today, we should strive for a world where everyone receive excellent, compassionate care from whomever they see. My hospital has a psychiatric ward for patients who speak Mandarin or Cantonese. This makes perfect sense. Beyond this, I don’t want hospitals to be segregated. I don’t know anyone who wants this.
Sixth, I agree that criticisms of “these issues” are not off-limits. Dr. Goldfarb’s often excoriates the Implicit Association Test and its purported ability to reveal bias, for example. This is entirely fair, and like any subject in medicine, scholarship on “these issues” needs to be held to high standards. Studies that purport to show racial bias in medicine are not sacrosanct, and those who reveal their methodological flaws are not racist. They are doing important work (here and here).
Finally, though Dr. Goldfarb may think he’s the only one to realize this, those who work in this space know the potential pitfalls of diversity education. For example, it’s no secret that:
Teaching “typical” characteristics of minority groups frequently promote stigmatization without promoting healthcare outcome improvements.
And though we may disagree on how to best get there, we all agree that “promoting healthcare outcome improvements” is the only thing the matters.
Areas of disagreement: These issues are rigorous medical science and doctors are not free of bias
Despite this, I disagree with Dr. Goldfarb’s central thesis. He claims that “teaching these issues is coming at the expense of rigorous training in medical science.” But “these issues” are rigorous medicine. Serious researchers have put a lot of work into this field and they’ve made important findings. However, Dr. Goldfarb implies that unless doctors are exonerated, investigations into the causes of racial disparities are not rigorous medical science. Of course, any study that begins with a predetermined conclusion is not rigorous medical science.
Moreover, contrary to what Dr. Goldfarb claims, there is evidence that doctors are not the only humans on the planet who are free of bias. Would that Black woman at LA County Hospital have received better pain control had she been a wealthy White celebrity in Beverly Hills? Of course. There’s no way she would have been left screaming that way. Though Dr. Goldfarb writes that “There’s no credible evidence that physicians are racist”, I’ve not seen him explain why Black patients receive less pain treatment than White patients. Instead, Dr. Goldfarb simply denies that this happens at all.
However, there is a vast literature on this and many similar topics. There are very few significant health metrics where the numbers are worse for White people than for any other race. Native Americans commit suicide at a higher rate, and Black overdose deaths exceeded White overdose deaths last year for the first time in 20 years, in part because White people received better treatment. Even Black children receive less pain control than White children. Things would be much better if all Americans received the same healthcare as most White Americans. Though the medical system is not solely responsible for these disparities, it is not the only American institution that is utterly without blemish today.
The evidence for this is literally everywhere. It turns out that a lot of medical students have some pretty strange beliefs, undermining Dr. Goldfarb’s claim that doctors are somehow immune to the same biases as everyone else. According to research by Dr. Janice A. Sabin racially biased beliefs do make their way into medicine:
“Black people’s nerve endings are less sensitive than white people’s.” “Black people’s skin is thicker than white people’s.” “Black people’s blood coagulates more quickly than white people’s.” These disturbing beliefs are not long-forgotten 19th-century relics. They are notions harbored by far too many medical students and residents as recently as 2016.
In fact, half of trainees surveyed held one or more such false beliefs, according to a study published in the Proceedings of the National Academies of Science. I find it shocking that 40% of first- and second-year medical students endorsed the belief that “black people’s skin is thicker than white people’s”.
What’s more, false ideas about black peoples’ experience of pain can lead to worrisome treatment disparities. In the 2016 study, for example, trainees who believed that black people are not as sensitive to pain as white people were less likely to treat black people’s pain appropriately.
This seems problematic. Maybe it would be good to identify these racist myths and disabuse students of them. To me, this seems to be an essential part of developing “individuals who know how to cure patients”.
Areas of disagreement: Learning about bias in medicine is not a personal attack on doctors
A central concern of Dr. Goldfarb is that learning about bias in medicine will hurt doctors’ feelings. He wrote:
Having talked with many physicians, I know that unwarranted accusations of racism are contributing to physician burnout and early retirement, making it harder for patients to receive care, especially in vulnerable communities.
A dishonest student once falsely accused me of not giving her time off to attend a relative’s funeral. So while some doctors may have faced unwarranted accusations of racism, I’m unaware of any evidence this is causing a mass exodus from our profession. I’ve not met anyone who is considering leaving medicine for this reason. Still, I doubt Dr. Goldfarb is not inventing such doctors out of whole cloth. The rules have changed, old jokes aren’t funny anymore, and no one likes to be accused of racism. However, too many doctors view any discussion of racism as a personal attack. This reveals more about the sensitivities of these doctors than anything else, and their delicate sensibilities are not a valid reason to ignore uncomfortable truths. Moreover, for every doctor who’s retired due to unwarranted accusations of racism, there are certainly others who’ve curtailed their ambitions or left the field because they experienced racism.
Of course, the purpose of discussing “these issues” shouldn’t be to make students feel guilty. Racial myths are part of our culture milieu, and a student who believes such a myth isn’t the moral equivalent of a KKK Imperial Wizard. Rather medical schools should need to teach about these myths to make sure no student graduates believing them. Additionally, there are doctors who’ve gotten away with horrible behavior simply because their actions weren’t filmed. Students should know these doctors are out there. Students should know they don’t want to make the news this way.
Dr. Goldfarb doesn’t see it this way. He said that anti-bias education “undermines the whole idea of a trusting physician-patient relationship”.
He doesn’t realize that “nonscientific factors” already determine what treatments many patients receive, and that this contributes to a lack of trust in our profession. He doesn’t grasp that patients might not feel the need to “seek out physicians matching their own race” if they knew that all doctors were cognizant of “these issues”. The idea that patients may perceive doctors’ behavior as racist even when he does not, does not seem to occur or matter to Dr. Goldfarb. He seems to feel patients and doctors alike should adopt his standard for what is and is not racist, and he sees no rational reason why a Black woman may prefer a Black obstetrician.
The reason, of course, is very simple – these women feel they will get better care with these doctors. It’s the exact same reason everyone chooses their doctor. Doctors should be dismayed that some patients have lost trust in some members of our profession. We can either pretend our patients have all been brainwashed by CRT or we can strive to earn their trust back. The choice shouldn’t be controversial. We should want all our patients to feel comfortable under our care. It may take some getting used to, but it’s less effort to try to call someone by their preferred pronouns than to write a book about why doing so is destroying American medicine. If it makes my patient feel more comfortable, I’ll do it. The only cost to me is a tacit admission that I value my patient’s self-identity.
Areas of disagreement: It’s OK to learn about problems we can’t solve, though learning about these issues is related to patient care
Another reason Dr. Goldfarb feels medical schools should not spend time on “these issues” is that doctors are often powerless to solve them. He wrote:
Above all, the medical profession should abandon the fantasy that physicians can be trained to solve the problems of poverty, food insecurity and racism. They have no clinical tools with which to address these issues.
He’s not entirely wrong. I’ve never handed an apartment key to a homeless person nor solved a patient’s food insecurity for longer than their hospital admission. There’s often very little we can do for people who have nothing. Moreover, not only do I not have the cure for racism, I will fail to convince many people that it can be a problem in medicine. Unfortunately, “these issues” do not have simple solutions or they would have been solved already. Dr. Goldfarb feels all doctors end up doing with “these issues” is referring the patient to a social worker, so there’s no reason for students to learn about them. They are literally someone else’s problem.
However, students spend a ton of time learning about problems they cannot solve. Does Dr. Goldfarb feel it’s useless for students to learn that the gene for Huntington’s disease is a CAG repeat on chromosome 4? Of course not. Have I helped a single patient by knowing that? Of course not. Like every doctor, my brain is filled with facts that are crucial to understanding the human body and disease, but have zero practical impact when actually treating patients. Instead of saying that students need not learn any such information, we should agree that it’s important for them to learn about all the factors that promote health and all the factors that are a detriment to it, even though we can’t fix everything.
Dr. Goldfarb believes that it’s inappropriate for any doctor to use a tool that isn’t a “clinical tool”, presumably a medication, procedure, or recommendation to exercise. Dr. Goldfarb gives no reason why doctors should so restrict themselves to these uses of “clinical tools”. Or perhaps we should drastically expand our concept of “clinical tools”.
Because while doctors can’t fix everything, we can occasionally fix some things, and this can make a big difference to some of our patients. Of course, “these issues” are not someone else’s problem. Anything that drastically affects the health of our patients is our problem whether we want it to be or not. We should use every tool at our disposal to help our patients. I discussed this previously in my essay on malingering when I wrote:
A referral to a domestic violence shelter or a suggestion that corrections officers transfer an endangered inmate to a safer jail unit can be life-altering. I wish I had been more attuned to this early in my career.
I learned all of this and more through my experience working at a large public hospital for nearly 20 years, and I’m not the only one who’s figured this sort of stuff out. Dozens of doctors Tweeted what they’ve learned under the hashtag #Goldfarbchallenge. You should definitely click on that link. You’ll why I said there are a lot of good doctors out there.
While much of the practical side of “these issues” must be learned through experience, learning about their importance should begin early in medical school. Virtually none of it was discussed in my training, which is too bad. I should have been encouraged to think about “these issues” from the start of my career, instead of having them treated like an afterthought. Again, there’s often very little we can do for patients, but I wonder what opportunities I’ve missed to help them over the years because I felt their social problems were not my problem. I wonder if Dr. Goldfarb is similarly curious about what opportunities he may have missed to help his patients.
Moreover, while medical school should focus on teaching students how to treat individual patients, it’s not wrong to get them thinking about the bigger picture. Yes, the fundamental goal of medical school is to train doctors who can treat women with obstetrical emergencies. But students should also be encouraged to think about why these tragedies are more common in Black women and what can be done to prevent them in the first place. I’m not saying every doctor must embrace only progressive policies as the only solution to societal problems (I don’t), nor must every doctor be an activist in their spare time. Tired doctors can do whatever they want when they get home from work. But it’s a good thing when someone is passionate about trying to fix problems in their community.
This sort of work should be appreciated and encouraged in our profession, because sometimes doctors’ actions matter. Circling back to menthol cigarettes, doctors spoke out against them, and the FDA is now considering a ban. While not every doctor must agree with this action, it shows that doctors are not destined to be powerless spectators regarding the factors that influence our patients’ health. And with recent news, our voices are needed now more than ever.
Additionally, it’s within the purview of medicine to formally study what interventions do and do not work for our most vulnerable patients. A recent study of intense medical and social outreach (healthcare hotspotting) to patients with “medically and socially complex conditions” failed to show any benefits, unfortunately. Dr. Goldfarb should appreciate that studying these interventions is as every bit “rigorous medicine” as studying a new medication. Hopefully an enterprising student will be motivated to come up with something that works.
Areas of disagreement: Learning about these issues won’t leave medical students unprepared to be doctors
Another of Dr. Goldfarb’s core worries is that learning about “these issues” will take an inordinate amount of time, preventing students from learning the enormous amount of material he feels they really need to know, such embryological development of the thyroglossal duct. He knows that time is a fixed commodity and believes that “Medical schools increasingly are preparing physicians for social activism at the expense of medical science”.
Like Dr. Goldfarb, I believe students should learn the basics, even if much of it is a rite of passage to be honest. But I’ve not seen him present evidence that learning about “these issues” is causing worse outcomes in patients as unprepared students transition into doctors. If he were right, we should have noticed deficiencies in young doctors by now. As I will explain, I think they’re better prepared than I was. They’ll know they’ll be graded on both their ability to identify the anconeus on a cadaver and whether or not they’re the type person who cares if their patient is in extreme pain.
Of course, students are smart enough to learn multiple things at once, and studying “these issues” need not take that much time. I imagine a couple lectures per month on “these issues”, as is done at my medical school, could convey an incredible amount of valuable information without leaving students deficient in the traditional curriculum.
Alternatively, “these issues” can be woven in lectures when appropriate. Many lectures make reference to heroic figures in medical history. That’s great, but it’s not wrong for a speaker to spend one minute informing students about people like J. Marion Sims, the “Father of Gynecology” who “conducted painful experimental surgeries on enslaved black women without using anesthesia”, though he used it with his White patients. “Lucy’s agony was extreme,” he wrote about one of these enslaved women. “She was much prostrated, and I thought that she was going to die”. A statue honoring this torturer was removed from NYC only four years ago. Though this dark history need not be discussed in every lecture, it shouldn’t be swept under the rug either. It may clue us into the fact that some of what we like to think of as ancient history is actually our present day reality.
“We in academic medicine have a duty to bring to light racist misinformation, stereotypes, and unconscious attitudes that contribute to disparities in patient care today.”
I may be projecting here, but I sense that on one level Dr. Goldfarb just doesn’t want to think about “these issues”, or he doesn’t want them to be true. So anytime anyone brings them up, they are dismissed as being “woke” or “promoting CRT” or some such buzzword that immediately signifies to his audience “don’t give this any serious thought“.
I can empathize with Dr. Goldfarb about this. I want to believe the best about my profession. Hearing unpleasant truths can make me feel uncomfortable and defensive. I certainly don’t want to believe I’ve ever undertreated a Black patient’s pain or mistreated anyone I work with. It’s nice to imagine that a Black inmate will receive the same care as a White titan of industry. But we know that’s not the case, and reality can’t be wished away by evidence-free moral panics that the future of American medicine is being sacrificed on the altar of wokeism.
While Dr. Goldfarb is pessimistic about the future capabilities of today’s students, I am not. Most students I meet understand that “these issues” are every bit as important to patient care as being able to identify Bowman’s capsule on a histology slide. They realize “these issues” are part of their curriculum for one reason only – to make them better doctors. They understand that a Black patient with a broken arm might fare better with a doctor who knows that Black patients feel pain just like White patients, than by a doctor who knows only that osteoclasts have multiple nuclei and a “foamy” cytosol. I believe these future doctors will have a greater understanding of the issues their patients face compared to when I was their age, and they’ve been able to teach me some important things. One even authored a book to help doctors identify clinical signs on darker skin. Though this is the only topic in medicine where skin color really matters, nothing like it existed until recently, in a profession Dr. Goldfarb believes to be free of racism.
Most students would agree with the conclusion of Dr. Sabin’s article:
As a nation, we must continue to reckon with the lingering history of racism in medicine. We in academic medicine have a duty to bring to light racist misinformation, stereotypes, and unconscious attitudes that contribute to disparities in patient care today. Dramatically reducing, and perhaps even eliminating, racial and ethnic disparities in pain treatment is an attainable goal — and a moral imperative.
Perhaps the OB resident I encountered in medical school would have cared about her patient’s screams had she recognized this. Maybe I would have spoken out if I had.

Deciding Which Risks to Take
No medical treatment is risk-free. Paul Offit’s new book covers the history of innovations that went awry and advises how to balance the risks of new medical innovations with the risk of not treating.
Harriet Hall on March 8, 2022
Tweet

It is always risky to use a new treatment before all the evidence is in, but it’s also risky to let a serious illness go untreated. Paul Offit’s latest book You Bet Your Life is about the “long and risky history” of medical innovations, giving readers much to consider while trying to decide which risk is greater.
Paul Offit is coinventor of a rotavirus vaccine that has been credited with saving the lives of hundreds of children every day. He is a practicing pediatrician at the Children’s Hospital of Philadelphia and a vaccine expert who has served on committees to advise the government. He has written several books on vaccines and now is branching out to address other subjects.
Medical breakthroughs are happening every day. They offer wonderful benefits, but as Offit says, “virtually every medical breakthrough has exacted a human price” and some people will surely die if they choose to wait for better evidence.
If we don’t learn from history, these tragedies will be repeated. In April 2020 the FDA granted an Emergency Use Authorization to hydroxychloroquine for the treatment of COVID-19, although “no study had ever found that it worked”. More studies showed that it didn’t work for COVID-19, that 10% of recipients suffered severe heart arrhythmias, and some of them died. The FDA had to withdraw its approval after only two months. With that in mind, many people refused to take the new coronavirus vaccines that had been developed at “Warp speed”. They thought they were avoiding risk, but their choice was not risk-free. It was a choice to risk infection, hospitalization, long-term disabilities, and death.
Offit takes us through the history of several medical innovations, including heart transplants, blood transfusions, anesthesia, biologicals, antibiotics, vaccines, x-rays, chemotherapy, and gene therapy. I thought I knew about the history of these innovations, but Offit fleshes out the story with details that were new to me.
Transplants
Most of the early heart transplants failed, but the patients “weren’t gambling with their lives, because their lives were at an end”. With the transplant, they died; but without the transplant, a certain death was imminent. Any risk seemed acceptable to the recipients. I knew effective transplants had to wait for technological advances and drugs that could prevent rejection (although they increased the risk of infections), but I hadn’t realized the importance of one impediment: there were no generally accepted criteria for brain death, and surgeons harvesting donor organs from a brain-dead patient could be tried for murder. Eventually, criteria for brain death were developed that were widely accepted and even supported by the Pope. But there are thousands of patients on the waiting list for a heart transplant, and a third of them will die while waiting. Offit hopes that genetic engineering may solve this problem by altering pig hearts so they can be used in humans.
Blood transfusions
The first three blood transfusions from animals (2 lambs and a calf) to humans were done in 1667 and they seemed to work. As Offit describes the second case: a drunken, middle-aged butcher was paid to receive blood from a lamb. After the procedure, he “jumped off the table, butchered the lamb, threw it over his shoulder, and ran off to a local bar to get drunk.”
The fourth patient died, and the surgeon was tried for murder. He narrowly escaped when the patient’s wife was executed for poisoning her husband. The publicity around this incident led to transfusions being declared illegal for the next two centuries; they were even banned by the Pope. After blood typing was discovered and the problems of clotting and infection were solved, transfusions became routine. A bit of trivia: in a paternity suit, blood typing proved that Charlie Chaplin was not the father of his mistress’ child, but since it was so new, blood type evidence was not admissible in court. A jury ruled that he was the child’s father, although he couldn’t possibly have been; and he had to pay child support and legal fees. Then, in the 1980s, Ryan White, a boy with hemophilia, developed AIDS from a blood transfusion and died. All Hell broke loose. Today, blood transfusions are handled with better safety precautions and better screening, and the risk is minimal but not zero.
Anesthesia
Remember John Snow, the doctor who identified the cause of a cholera epidemic and stopped it by removing the handle of the Broad Street pump? I was surprised to learn that he was the same doctor who gave Queen Victoria chloroform for childbirth, which convinced many other women to accept pain relief for labor. Unfortunately, chloroform proved more dangerous than ether or nitrous oxide; but it continued to be used for decades before it was replaced by safer anesthetics.
Tetanus from diphtheria antitoxin serum
The blood of a horse named Jim had been successfully used to supply many batches of the antitoxin that effectively treated diphtheria. Remember Balto, the sled dog who rushed the lifesaving antitoxin to Nome? When Jim developed tetanus, his blood could no longer be used, and the lab personnel were instructed to discard his blood samples. Human error by a janitor resulted in the tetanus-infected blood being mixed with the safe blood, and 13 children in St. Louis died unnecessarily. There were other innocent victims. People elsewhere were frightened by the disaster in St. Louis and refused treatment: the death rate in Chicago increased by 30%.
The Elixir sulfanilamide disaster
Prontosil was one of the first antibiotics used to treat humans. Gerhard Domagk was awarded a Nobel Prize for discovering its effectiveness, but Hitler refused to let him travel to Sweden to receive it. The Massengill company dissolved the active ingredient sulfanilamide in diethylene glycol to make a convenient liquid preparation for children. They didn’t test it for safety. By the time they realized diethylene glycol was a poison related to antifreeze, 105 people had died. The Food, Drug, and Cosmetics Act was passed to prevent similar disasters, but the Massengill company was never held accountable. It was only fined $26,000 for misbranding and had to pay $148,000 to settle damage claims.
Polio vaccine
The Salk polio vaccine saved countless lives, but it also caused a man-made polio epidemic with tens of thousands of cases of paralysis, hundreds of them permanent, and 10 deaths. Offit asks, “Why don’t we know this story?” The Salk vaccine was not the first; two earlier polio vaccines were developed in 1934 and 1935. Both led to paralysis and were withdrawn. Offit explains what happened and why 20 years passed before anyone dared try again.
Salk’s vaccine worked, but one manufacturer, the Cutter company, had to recall their product after two specific lots had paralyzed 51 children and killed 5, and 74 unvaccinated family members of vaccinated children had been paralyzed. In total, 70,000 people had developed polio and ten had died. Offit explains why: Salk had made a false assumption about the straight-line virus inactivation curve, Cutter had used an inferior glass filter, safety tests were “woefully inadequate”, they let the filtered virus sit in a refrigerator for weeks before inactivating it with formaldehyde, and they never constructed a graph to prove that formaldehyde was killing the virus reproducibly and in a straight line. Offit calls this the Scale-Up Incident; it illustrates some of the things that can go wrong when new vaccines are mass-produced by different companies.
The vaccination campaign switched to the Sabin oral vaccine. It was effective and easy to administer, but it could mutate back to a virus that resembled natural poliovirus. This only occurs in one of every 2.4 million doses, but in 2000 the US switched back to the safer Salk inactivated vaccine.
X-rays
X-rays were an incredibly useful aid to diagnosis, but radiologists were slow to appreciate the risks of radiation. Offit tells of a 1920 professional gathering of radiologists “where so many attendees were missing hands and fingers that when the chicken dinner was served no one could cut their meat”! As a child, I remember seeing x-ray machines in shoe stores to assist in shoe fittings; they weren’t banned until the 1970s.
Other risks
Offit goes on to cover the history of other potentially risky innovations, including chemotherapy and gene therapy. He illustrates these with fascinating stories. In 1946, a 52-year-old man developed hoarseness, eye pain, and inability to speak or swallow; the cause was a large tumor at the base of his skull. Conventional treatment with radiation was not helpful. After treatment with a new chemotherapy drug, a folic acid antagonist, “the tumor shrank and he gained back most of the eighty pounds he had lost. By 1948, he was well enough to put on his old New York. Yankees uniform – the one with the number 3 on the back – and attend the 25thanniversary of the opening of Yankee Stadium. His name was Babe Ruth.”
Risk is unavoidable
Offit stresses that no treatment is risk-free. He says:
…people should make the most informed, most clear-eyed, most dispassionate decisions using all of the information available, knowing that decisions under uncertainty might be the wrong ones. Every choice involves risk, even the simplest ones.
Offit ends the book with a chapter on living with uncertainty. If we don’t learn from the disasters of history, they are sure to be repeated. Unanticipated tragedies are unpreventable. Some people think that when someone is sick or dying, any therapy is worth a shot. But Offit reminds us that “if a medicine doesn’t work, it can only do nothing or hurt”. Animal testing can be falsely reassuring, since “mice lie and monkeys exaggerate”.
Conclusion: A very worthwhile book
Offit’s research is thorough and impeccable. He writes well, and he’s a great storyteller.
Reading this book will inform you and entertain you, and it will give you much to think about for your future decisions about innovations in medical technology. I highly recommend it.

If You Think Doctors Don’t Do Prevention, Think Again
Harriet Hall on August 9, 2016
Link


One of the common criticisms we hear from alternative and integrative medicine proponents is that doctors don’t do anything to prevent illnesses and have no interest in prevention. They claim that doctors are only trained to hand out pills to treat existing illnesses. Sometimes they even accuse them of deliberately covering up cures and wanting to perpetuate illnesses like cancer so they can make more money by treating patients. Nothing could be more absurd. Every reputable doctor would rather prevent illnesses than treat them. In his book Heart 411, cardiologist Steven Nissen even said he would be glad to see his specialty become obsolete: “Don’t worry about us; we will gladly hang up our scalpel and stethoscope if we can find a better way to lead you to a heart-healthy life.”
Doctors own prevention. They invented it, from vaccines to clean water to preventive screening tests. Mainstream medicine was responsible for the greatest preventive achievement in history: the smallpox vaccine campaign succeeded in preventing anyone from ever getting smallpox again. I defy you to comb through historical records and find any doctor who ever said “Let’s stop vaccinating for smallpox so we can make more money treating its victims.”
Prevention is one of the six fundamental principles of naturopathy. Alternative practitioners pride themselves on prevention, but they don’t actually do a very good job of it. In fact, there is evidence that their patients are less likely to get immunizations and some of the standard preventive screening tests recommended by the USPSTF. Instead of rigorously implementing evidence-based preventive strategies, they tend to offer other speculative, untested recommendations.
Preventing Zika virus infections
The current Zika crisis is a case in point. Doctors are doing prevention. Mainstream medicine has responded with mosquito eradication programs, effective mosquito repellents, travel advisories, and public information campaigns. Two experimental vaccines have already been proven effective in mice, and human trials are already underway.
Meanwhile, what are the CAM folks doing? Recommending things like these:
    • Treating clothes with essential oils.
    • Natural mosquito repellents like lemon eucalyptus oil, lavender and garlic.
    • Eating vegetables rich in vitamins A and C and taking a daily Arogyam Pure Herbs Madhuparni tablet.
    • Home remedies that they claim will cure Zika infection naturally:
        ◦ Diatomaceous earth or clay.
        ◦ Homeopathic Eupatorium Perfoliatum
        ◦ Papaya leaf juice
Mike Adams of Natural News warns:
If the Zika virus spreads into a devastating pandemic, it will only be because natural cures that already exist in nature are being systematically suppressed or criminalized by our corrupt, criminally-run medical police state system.
No, I don’t think so!
Bottom line: doctors are doing real prevention; non-doctors are doing bogus prevention.
The medical literature
Anyone who thinks doctors don’t do prevention must not be reading medical journals. Much of the medical literature addresses prevention rather than treatment. A recent review article in the American Family Physician, the journal of the American Academy of Family Physicians, is a good example. The title is “Diet and Physical Activity for Cardiovascular Disease Prevention” and its six pages and 36 references contain not a single mention of statins or any other drugs. It provides evidence-based recommendations to follow a DASH or Mediterranean diet and to get at least 150 minutes of moderate-intensity aerobic activity, 75 minutes of vigorous-intensity aerobic activity, or an equivalent combination each week.
What do naturopaths recommend for cardiovascular disease prevention? They may make the same evidence-based diet and exercise recommendations, but they also recommend non-evidence-based interventions like avoiding dairy, eating organic whole foods, taking vitamins and arginine, and using herbal dietary supplements like gingko and hawthorn. Some recommend CoQ10, omega-3 fatty acids, green tea, pomegranate, magnesium and potassium. And others add cinnamon, ALA, lutein, and plant sterols.
There are pragmatic studies claiming that the addition of naturopathic care to standard care decreases cardiovascular mortality, but pragmatic studies can’t determine effectiveness. Many of those studies are flawed and not properly controlled. They fail to distinguish between the effects of intensive counseling and the effects of the natural remedies they added. And those natural remedies are all over the map, so it is impossible to meaningfully compare studies or to determine which elements actually worked. I have seen no evidence that a patient under the care of a typical naturopath is any less likely to have a heart attack than the patient of a typical mainstream clinician.
What the critics really mean
When they claim “doctors don’t do prevention,” what they really mean is that they don’t think doctors spend enough time advising patients about things like exercise and diet. They seem to think doctors should take a detailed diet history from each patient and provide detailed personalized recommendations for what to eat at each meal. But that is not the job of doctors; that is the job of dietitians. Doctors understand the principles of good nutrition and provide general advice, but refer patients to dietitians for the nitty-gritty details.
Critics often have an exaggerated idea of what diet and lifestyle measures can accomplish. Only about a third of cancers are preventable by diet. Some people believe that if you eat right you can’t possibly get sick, and a recent commenter on this blog claimed that even genetic diseases could be reversed in three years by proper diet! They are wrong.
The critics cite anecdotes about patients who say they were given prescriptions with nary a word about preventive measures: for instance, statins to prevent cardiovascular events but no advice about weight loss, exercise, or heart-healthy diet. That may happen, but I suspect it is the exception rather than the rule.
I question the reliability of patients who claim their doctor never once mentioned smoking, weight loss, exercise, or diet. Memory is fallible. Studies have shown that 40-80% of medical information provided by healthcare practitioners is forgotten immediately, and half of the information that is remembered is incorrect. And surely people are less likely to remember advice that they didn’t want to hear.
The official guidelines for statins have been widely criticized, but the critics rarely acknowledge this paragraph from the report:
It must be emphasized that lifestyle modification (i.e., adhering to a heart healthy diet, regular exercise habits, avoidance of tobacco products, and maintenance of a healthy weight) remains a critical component of health promotion and ASCVD risk reduction, both prior to and in concert with the use of cholesterol-lowering drug therapies.
We can do better
Instead of “doctors don’t do prevention,” a more valid criticism would be “doctors don’t do as good a job of prevention as they might.” That is something doctors acknowledge and are already constantly working on, looking for effective ways of getting patients to understand and retain information and effective ways of improving compliance. And doctors don’t carry the responsibility for prevention alone. As with horses, doctors can lead patients to water but can’t make them drink. Doctors can encourage their patients to exercise but they can’t stand over them with a whip and make them get up off the couch. Doctors can lead patients to a healthy diet but they can’t follow them around 24/7 and slap their hand away if they reach for a slice of pecan pie instead of a piece of fruit.
Saying doctors don’t do prevention is a vile, ignorant canard. Showing how they could do better would be helpful, constructive advice.

The Primum non nocere principle in psychotherapy: A science based approach
Is the principle of primum non nocere, to do no harm, applied by psychotherapists?
Tomasz Witkowski on July 26, 2018
Link

[Editor’s note: With Scott away on vacation, this Thursday we bring you the inaugural post of Tomasz Witkowski, a psychologist, academic, founder of the Polish Skeptics Club, and author of two books favorably reviewed by Harriet. Welcome Tomasz!]
First, do no harm – a rule that enjoys practically unconditional acceptance by all of human civilization, at least when it concerns healthcare interventions. Students of medicine have this hammered into their heads beginning with their first lesson. It is so universal that we treat it as a part of the Hippocratic oath, while it in fact never was one; Hippocrates’ authorship itself is also frequently called into question.
If we consider the matter more carefully, it turns out to be not only inhumanly difficult to adhere to, but even cruel in a certain sense. A doctor, a physiotherapist, or a nurse is faced with a person who is suffering, in need of help, sometimes immediately; but before that help is given, it is necessary to take into consideration all of the surrounding circumstances and professional knowledge in order to be entirely sure that no harm will be done. And if they hesitate? If they aren’t completely sure? Should they refrain from intervening? The rule is clear – yes. Not doing harm takes priority over doing something which may be helpful, but which also bears too great a risk of harm. Thus, sometimes the principle of primum non nocere means we should refrain from providing assistance to a person who is suffering. It means that not-acting is better than rash, uncertain action.
And yet, when taking a closer look at medical practices, it turns out that this overarching principle is giving way to one that stands in opposition – melius anceps remedium quam nullum – an uncertain treatment is better than no treatment at all. Those who give care prefer to do anything at all rather than nothing. And what if it brings relief in suffering? The motives of the people inclined to follow the latter rule are diverse. They may attempt to do something out of pure sympathy, or due to pressure put on them by the patient. They may be pushed into doing something by their environment, the demands of the sick person’s family, or even their superiors. Sometimes they are driven by aversion to the helplessness which they feel when refraining from action, or by the need to verify their own competences and a feeling of their own professional value. Not infrequently, they are also motivated by economic considerations – after all, patients prefer to return to doctors or therapists who have done something, prescribed some medicine or performed some intervention, rather than to those who simply shrug their shoulders and say there’s no tried-and-true method, so the rule of primum non nocere means that nothing is to be done.
An unequivocally negative and categoric judgement of submission to the principle of melius anceps remedium quam nullum would be nothing but a manifestation of moral absolutism, which is not something to which I am prone. A fair assessment of such conduct cannot be offered without taking into account motive on the one hand, and consequences on the other. This is expressed in the risk-benefit ratio, a measure broadly accepted in medical studies. Rather than refraining from acting in the face of even the smallest risk, we calculate the relation of the risk of harm to the potential benefits that the patient may receive, silently accepting the tenets of utilitarianism.
Psychotherapy and primum non nocere
And yet, when I observe the practice of medicine, particularly psychotherapy, I am unable to avoid the impression that many of those offering their services in medicine and psychotherapy have simply forgotten about the existence of the principle of primum non nocere, and their decisions have nothing in common with a rational calculation of the threats and benefits. But is the application of ineffective therapeutic methods consistent with that principle? The cautious answer is yes – as long as they bring positive psychological effects comparable to those that come from the use of a placebo. Yet this cautious answer most certainly requires elaboration. Yes, but only when we are entirely certain that effectiveness was measured in comparison to a placebo group, and not to a control group that was not subjected to any treatments. Furthermore, if this condition is fulfilled as well, we should also obtain the unshakeable conviction that there are no methods in existence that would be more beneficial in the very same situation the patient is in. Indeed, the suffering of a patient treated improperly may deepen, and that patient, firm in the belief that the treatment has been successful, will not seek more beneficial methods; this is fuel for needless suffering.
The issue of ineffective treatment should also be viewed from the perspective of the difficulties the patient goes through. This frequently involves abandoning other activities (family, social, professional, etc.) to devote time to treatment. Patients devote a portion of their financial resources to treatment. That alone is already a difficulty. It becomes an even greater burden when such therapy is offered to terminally ill patients – rather than live out their final days in the company of loved ones, doing the things that give their lives sense, they participate in empty rituals of ineffective therapies. I would thus submit that ineffective treatment is equivalent to the infliction of harm on a patient, and I will attempt to demonstrate how common this phenomenon is in psychotherapy.
In 2017, a team directed by Evangelos Evangelou from the University of Ioannina in Greece did something out of the ordinary – a meta-analysis of meta-analyses concerning the effectiveness of psychotherapy. Let us recall that meta-analysis is a method that gathers in one place the results of studies from many experimental works. A meta-analysis can only include those experiments that include a control group and in which random assignment of participants to groups was employed, and thus studies meeting the highest methodological standards. What Evangelos Evangelou and his team did was to gather 247 finished meta-analyses and combined them into one single meta-analysis. The result was striking. While the majority of meta-analyses favoured psychotherapy as an intervention method, a mere 7% of them contained convincing evidence that psychotherapy was effective! Does the remaining 93% guarantee that we will remain faithful to the rule of primum non nocere?
At present it is estimate that there are over 600 different modalities (approaches, schools) of psychotherapy practiced around the world. Can the average psychotherapist, deciding on the choice of a specific therapeutic intervention, be sure that it is the very best choice for the patient? Well, I am afraid that none of them (or at least nobody I have met) boast the brainpower to study, remember, and ultimately engage in measured reflection on the benefits and threats of them all. And what about those who are creating new modalities? Do they begin with an analysis of all existing ones, arriving at the conclusion that one more is necessary for improving the effectiveness of psychotherapy? If this were the case, it would mean that every new modality is better than the previous one, and the vintage ones would be slated for extinction. But this is not happening – the majority of them are doing quite well, and rather than decline, their numbers are growing. In such conditions is it really possible to implement the principle of primum non nocere?
The consequences of the existence of so many ineffective interventions are borne by patients. Therapists continue to benefit from them. Patients subjected to ineffective interventions become the object of a sort of “recycling”. Disappointed by the absence of effects, they reach successive therapists who take a portion of their resources, and if those turn out to be ineffective, there is a chance that still other representatives will get the chance to demonstrate their effectiveness. “Record holders” that I have encountered in my life have been in psychotherapy for over 25 years, and I’ve heard of a single patient who was treated in succession by as many as 18 psychotherapists. Primum no nocere or perpetuum mobile?
The issue of faithfulness to the principle that has become the title of this essay stands out even more when we shift our perspective from the quite common absence of effectiveness towards the negative effects of treatment undertaken. In medicine this is the well-understood – but still shameful – phenomenon of iatrogenesis. In the United States alone, mistakes in treatment cause between 230,000 and 248,000 deaths each year. Of those, around 12,000 are caused by unnecessary surgical interventions, 27,000 by mistakes made in hospitals, 80,000 by hospital infections, and around 107,000 people die as a result of side effects from medicines applied properly. And how does the situation look in psychotherapy?
To tell the truth, it looks a lot better, because psychotherapists and researchers have done their best to ensure that as few such numbers as possible have seen the light of day. Literally. In 2014, an article was published whose authors had reviewed all studies on the effectiveness of psychotherapy published in 2010. From among the 132 works they gathered, a mere 28 of them (21%) contained any information that would suggest their authors had controlled the negative effects of the examined therapeutic interventions. Only four of them (3%) contained descriptions of those effects and methodologies for collecting data. Five of them did contain a description of negative effects, but there was no information about the manner in which they were measured. In four additional reports (3%) it was only briefly stated that no undesirable effects had occurred, while 15 studies (11%) supplied merely information about declines in patients’ condition or made mention of the fact that such a decline was being monitored. The remaining 104 papers (79%) contained no information about the monitoring of negative effects of psychotherapy.
Unfortunately, these results are not unique. Conor Duggan and collaborators analysed 82 psychotherapeutic procedures confirmed by the National Institute for Health Research in the United Kingdom in the period 1995-2013. None of them mentioned potential negative effects of psychotherapy, although the majority of pharmacological therapy protocols did.
Knowledge on the negative effects of psychotherapy is also absent among American psychologists. In 2006, Charles Boisvert from the Rhode Island Centre for Cognitive Behavioral Therapy and David Faust of Brown University Medical School conducted a survey among 181 psychologists and APA members operating their private practices across the United States. As many as 28% of that group was unaware of the existence of negative effects of psychotherapy. In turn, other studies have shown that therapists frequently assess the effects of conducted psychotherapy inaccurately – they regard it as effective even when it does not generate any positive results. As the article’s authors warn us, this involves the risk of continuation of ineffective therapy, and in consequence a worsening of the patient’s condition. This is confirmed in studies by Michael Lambert and his collaborators, who asked 40 clinicians to make predictions on the potential for negative effects to occur at the conclusion of psychotherapy among 550 of their patients. It turned out that even despite information that the general proportion of patients experiencing negative effects was around 8%, they predicted that virtually no negative effects could occur among their patients. They only indicated three cases in which the patient’s condition could decline. The prediction was accurate in one case. The clinicians failed to identify 39 other patients whose condition deteriorated in the course of therapy.
While the issue of iatrogenesis in medicine is studied systematically – a mere browsing of the literature shows a range of books and monographs on the subject – I have found a mere handful of articles on the subject of iatrogenesis in psychotherapy, the majority of which were theoretical. I have not found a single book dedicated in its entirety to the subject, at least not in English.
When I look at this data, I am convinced that the principle of primum non nocere, allowing us to identify threats to patients in psychotherapy, has been replaced by the suggestion made frequently by contemporary youngsters – “stay out of it!”
The most well-known public declaration of adherence to the principle of melius anceps remedium quam nullum was a declaration in 1891 by the Swiss psychiatrist Gottlieb Burckhardt, who said:
Doctors are different by nature. One kind adheres to the old principle: first, do no harm (primum non nocere); the other one says: it is better to do something than do nothing (melius anceps remedium quam nullum). I certainly belong to the second category.
It evoked outrage among his contemporaries, and Burckhard’s career ended in ridicule and indignity. But his attitude found many admirers, particularly among neurosurgeons at the end of the 19th and in the first half of the 20th century. While they did not flaunt it, they did follow the advice of Burckhard, infamous proof of which can be found in the thousands of victims of lobotomies and other, often haphazard, surgical interventions performed on psychiatric patients.
Equally famous to Burckhardt’s declaration is the riposte of the French psychiatrist Armand Semelaigne, who declared that “an absence of treatment was better than a bad treatment,” thus demonstrating the fundamental meaning of the principle primum non nocere. Where is psychotherapy to be found among these ethical precepts? Is it following the simple path marked out by Armand Semelaigne? Or rather seeking possibilities for action in the footsteps of Burckhard?

Illness, Healing, and Other Terms That Can Be Confusing
Watch your language. Some terms that are frequently confused: pain vs. suffering, disease vs. illness, cure vs. healing. Science-based medicine focuses on diseases; alternative medicine focuses on illness: that explains a lot.
Harriet Hall on November 16, 2021
Link

Watch your language! Words matter. Some medical terms are frequently confused: disease vs. illness, pain vs. suffering, cure vs. healing. Precision of language improves precision of thought. The following is an article I wrote for my “Reality Is the Best Medicine” column in Skeptical Inquirer. I thought these points were important enough to share here.
Disease or illness?
The words “disease” and “illness” are often used interchangeably, but their meanings are very different. Diseases are biomedical entities where there is impairment of the normal functioning of the body. Illness refers to the way a patient experiences a sickness. There can be disease without illness and illness without disease.
Diseases may have external or internal causes. COVID-19 is an infectious disease that occurs when a microorganism in the external environment, the pandemic coronavirus, takes up residence in the body. Diabetes is an internal dysfunction where the body no longer produces enough insulin. There are many categories of disease such as infectious diseases, hereditary diseases, deficiency diseases, and physiologic diseases. Diseases can be classified as congenital or acquired, acute or chronic, iatrogenic (caused by medical treatment) or idiopathic (cause not known). Terminal diseases are those expected to end in death. AIDS was once considered terminal, and it is still not curable; but it has become a manageable disease that patients can live with and that may not even decrease their life expectancy. When the cause of a disease is not understood, it is sometimes referred to as a syndrome or a disorder.
Patients can have a disease and not know it. When patients test positive for COVID-19, many of them are asymptomatic. They have the disease but no symptoms that would make them feel ill. A heart attack may be the first sign of a long-standing cardiovascular disease. The coronary arteries have become increasingly narrowed by atherosclerotic plaques over the course of many years, but the patient remains unaware until a complete blockage or a ruptured plaque deprives heart muscle cells of the essential blood supply they need to function. Only then do the cells begin to die, and the patient experiences chest pain. Conversely, patients may feel sick and be suffering from an illness when there is no actual impairment of biologic functions. One example is hypochondria.
“Dis-ease” is an imprecise term often used in complementary and alternative medicine (CAM) to indicate a vague feeling that all is not well, or to suggest that some optimum condition of “ease” and perfect health is lacking and can only be attained by following the advice of the CAM practitioner.
“Illness” refers to the personal experience of suffering, but it is also a social experience conforming to the norms of society and influenced by culture, education, beliefs, and expectations. Diseases can kill, but illnesses cause most of human suffering through imagination. CAM and mainstream medicine are often at odds because they don’t speak the same language. CAM doesn’t treat disease; it treats illness and cares more about how an individual patient feels than about scientific evidence. It relies heavily on suggestion and placebo effects. CAM sustains hope. Everyone resists the idea that there’s nothing more that can be done. It’s not acceptable to “do nothing,” even when doing nothing is the safest and most reasonable approach. Scientific medicine is often perceived as impersonal, mechanical, uncaring, and as destroying hope. Taking an ineffective CAM remedy lets patients believe they are “doing something.” Patients can benefit from the illusion that they are in control of their destiny and from the positive and caring interaction with another human being.
Cure vs. Healing
Cures are rarely achieved, but healing is always possible. The role of doctors is “To cure sometimes, to relieve often, to comfort always”, an adage that originated in the 1800s with Dr. Edward Trudeau, the founder of a tuberculosis sanatorium (back then, there was no cure for TB, so relief and comfort were all he could offer). In some cases, doctors claim to have cured (for instance, by excising a tumor or removing a foreign body); but in most cases it is really the body that cures the disease, and the doctor’s role is limited to facilitating that process.
“Healing” means different things to different people. Wounds may be due to trauma or surgical incisions, and it is usually obvious when they are healing. The healing process is complex but fairly well understood. First there is an inflammatory phase with a clotting cascade, a response of blood vessels, and the arrival of white blood cells. Then there is a proliferative phase where new tissues replace the damaged tissues. Then maturation and remodeling occur. The healing of a broken bone can be verified by x-ray.
A different meaning for “healing” was brought home to me when I watched a documentary about João de Deus (John of God), a psychic surgeon in Brazil, a charlatan who is currently in prison for sexually abusing 600 women, including his own daughter. They interviewed a woman in Italy who had travelled to Brazil to be treated by him and who firmly believed he had healed her. She had advanced metastatic breast cancer and had stopped the conventional treatments of her oncologist when she decided to see the psychic surgeon. She died in 2003. At the time of the interview, her disease was progressing, and she was dying of cancer, yet she claimed to have been healed. Was she in denial? Didn’t she realize the cancer was killing her? What was she thinking? Was this a triumph of hope over reality?
I eventually understood that by “healing,” some people mean a psychological process where they have come to terms with their diagnosis. They have achieved acceptance and are no longer distressed. An article in the Annals of Family Medicine explains:
Healing was associated with themes of wholeness, narrative, and spirituality. Healing is an intensely personal, subjective experience involving a reconciliation of the meaning an individual ascribes to distressing events with his or her perception of wholeness as a person.
Their conclusion: “Healing may be operationally defined as the personal experience of the transcendence of suffering”. OK, but measuring that personal experience is problematic. That concept of healing is hard to pin down, and it relies on patient self-reporting. It is more of a metaphysical concept than a scientific one.
Pain vs. suffering
There is no convenient objective way to measure pain. We can only accept a patient’s report that they are experiencing pain. We ask them to rate their pain on a scale of one to ten, where ten is the worst pain, zero means no pain, and anything over 8 is considered severe pain. I hate it when I am asked to rate my pain on that scale; it lacks nuance. I never know what to say, and my answer will vary depending on my mood. What should I answer if I am pain-free at the moment, but the pain was bad enough to keep me from sleeping last night? What if I wouldn’t call it pain but simply discomfort? What if a patient is a stoic who never complains? What if the patient is a person who habitually exaggerates or imagines symptoms? What if they report a pain level of 15 out of ten? A good clinician can investigate by asking questions like “Do your teeth itch? Do your stools glow in the dark?” If patients answer yes to all such questions, the doctor will know to discount their reports as unreliable.
Pain is a physical sensation; suffering is a psychological one. Pain can occur without suffering, as in the famous movie scene where Lawrence of Arabia extinguishes a match with his bare fingers and explains that the secret is not minding that it hurts. And suffering can occur without pain. Anticipation of pain often causes more suffering than the pain itself. “Pain and suffering” is a legal term used in the courts to help determine monetary compensation for injuries.
Suffering is determined by the sufferer’s thoughts. People try to interpret the meaning of their pain. For Lawrence of Arabia, the pain in his fingers meant an opportunity to impress other people. Can you expect the pain to be over soon? Will it soon stop with a desirable outcome like the birth of a baby? Does the pain mean you are going to die? Can you expect the pain to get worse?
Veterinarians claim to know when an animal is suffering, but they don’t have access to the animal’s thoughts. They are limited to observing the animal’s behavior. If an animal stops eating and is less active, what is going on in their mind? Do they experience suffering, or do they merely feel the experience of not wanting to eat or move? Animals live in the moment, not in the past or future.
For that matter, is a crying baby experiencing suffering, pain, or just frustration? Is crying a way to manipulate adults into satisfying their needs with a diaper change or a feeding? Animals and newborn babies don’t have the mental development to think like adult humans. Newborns are incapable of obsessing about past instances of pain or anticipating future suffering. They are incapable of speculating about what the pain might mean. A new baby may cry when blood is drawn, but they also might cry just as vigorously when being restrained or having their clothing changed.
When I was an intern, two patients on our surgical ward had the same operation (inguinal hernia repair). One was an adult who continued to complain of pain for days and refused to get out of bed; the other was a toddler who was happily running around the ward right after the surgery. The toddler had no expectation that he would be incapacitated by pain; the adult was unduly worried and avoided doing anything he imagined might cause more pain.
Hypnosis could be thought of as “mind over matter.” It can reduce the experience of pain through selective attention/inattention; the suggestions of the hypnotist direct the patient’s thinking. Similarly, natural childbirth utilizes thought control and distraction to reduce the need for anesthesia or drugs.
Things to remember
Words matter. Improving the precision of our language should improve the clarity of our thinking.
Science-based medicine treats diseases, CAM treats illnesses. (This explains a lot!)
Healing is an elastic term that may not mean what you thought it did.
        ◦ Suffering is not the same as pain. Each can occur without the other.

For Discussion: Should I Only Write About Fake Stuff If It Is Well-Known?
Does writing about questionable topics that are not well-known do more harm or good? There are arguments on both sides.
Harriet Hall on March 19, 2019
Link

A person who describes himself as a “pseudoscience fighter” e-mailed me to chastise me for writing about Prodovite last week. He felt compelled to offer me some advice. He made some good points; but they were things I had already thought about, and they didn’t change my mind. I thought it might be useful to open up the discussion and see what others think.
He praised SBM for its approach to pseudoscience and falsehoods that are well-known or well-publicized; but he argued that if the fake stuff is not really well-known, it is best to quash it silently (by ignoring it). He said he had seen thousands of scams similar to Prodovite, but not Prodovite itself; he would not have known about it if I hadn’t written about it. So he thinks I am providing free publicity and may inadvertently influence people to try it. He mentions the Streisand effect and the danger that some anti-scientists will post rebuttals on their own blogs.
He says that by telling the world that I responded to an email I may encourage scammers to write me, get me to critique their offerings, get me to provide free advertising, and then decry my harshness and negativity as a rejection of things science can’t understand. He says granting a response implies there is worth in investigating such junk. And critiquing bad studies is counterproductive because readers will learn that those studies exist and will cite them as “evidence.”
As I usually do, I ended the Prodovite article by pointing out that it’s impossible to know whether a product is safe and effective if it has never been properly tested. He thinks scammers will pick on this and say it shows I’m actually not really sure that Prodovite does not work. Well, I’m not. And the sellers and customers can’t really be sure that it does work. I think that’s an important point to make, even if some readers fail to understand.
He urges me to make my assertions as precise and uncompromising as possible. I try not to make dogmatic assertions because they are seldom warranted by the evidence. It may be impressive to assert “vaccines have been proven not to cause autism” but if I wrote that it would discredit me and make people who understand science discount the rest of what I wrote. It’s more precise and accurate to say, “Scientists have studied this extensively and have found no evidence that vaccines cause autism. In fact, the evidence doesn’t even show a correlation.”
He didn’t mention one problem that I worry about: the backfire effect, where confronting believers with evidence against their belief only makes them believe even more strongly and encourages them to think up more rationalizations to justify their confirmation bias.
I had not heard of Prodovite before, and I didn’t want to waste my time writing about a product that only a handful of people had heard of. How am I to judge whether a product is well-known? I tried to get some idea by searching online. I did a Google search for “Prodovite” and got 102,000 results. It seemed obvious to me that it was fairly well known. Is 102,000 not enough? How many are enough to consider it well known?
The results were all favorable information and testimonials. There was no negative information, no skeptical comments, no application of critical thinking skills. That’s the problem with dietary supplements like this, and with CAM treatments in general: most scientists don’t take them seriously enough to bother investigating them or commenting on them (hence the need for the Science-Based Medicine blog).
My search told a one-sided story. I couldn’t find a single website questioning the claims for Prodovite. Not a single comment about the nonsensical claims in the advertising (like the “molecule” they observed that was larger than the blood cells, or the abnormally small white blood cells that quadrupled in size in a mere five minutes). Not a single criticism of the only clinical study, no hint that it was junk science. No criticism of their choosing bogus live cell analysis for a “scientific” test.
Now, when potential customers search the Internet, they will find my article in the first page of results. They may ignore it, but at least now there is the possibility that inquiring minds will learn that Prodovite is not universally accepted as a superior, science-based nutritional supplement. I think it’s worthwhile putting that information out there. My SkepDoc’s Rule is “before you believe a claim, find out who disagrees with it and why.” And you can’t do that if no one has bothered to disagree with it yet.
Other articles I have written are ranking high on Google searches and I know they have reached a lot of people because many of them have contacted me to thank me, to ask questions, to tell me about other studies that they think constitute proof (they don’t!), or to offer testimonials about their own good or bad experiences.
Sometimes I can tell I have really hit a nerve and upset the true believers. It doesn’t change their minds, but it may worry them a bit. I can hope I have planted a tiny seed of doubt that may be fertilized by information from other sources over time and may eventually bear fruit. I once got an email from an acupuncturist who had stopped acupuncturing and found another career because of what I wrote. I said it wasn’t supported by the evidence. He “knew” it was and set out to find the evidence and prove me wrong. Instead, he proved himself wrong. In fact, he said one of the textbooks he had relied on had no references to supporting studies at all – just bald assertions.
I still regularly get emails from Protandim distributors every time a new study comes out. The response to my Isagenix article was so hilarious it generated another article.
Sure, there are thousands of similar products. I couldn’t possibly write about all of them. My hope is to educate readers and give them ideas to help them evaluate similar products for themselves. My goal is to encourage readers to ask questions, to teach them how to think, not what to think.
Does writing about things like Prodovite do more harm than good? I don’t think so. Your comments are welcome.

The AMA’s Guide to Politically Correct Language: Advancing Health Equity
The AMA urges us to promote health equity by adding progressive political language to what were once simple statements of fact.
Harriet Hall on November 2, 2021
Link

The American Medical Association has just released Advancing Health Equity: A Guide to Language, Narrative and Concepts. Jesse Singal has been tweeting about it, calling it “a strange document that calls for doctors to insert progressive politics into even plain statements of fact.”
Some of the most egregious language can be seen in Table 5:While this is all true, I think such language is too cumbersome, convoluted, confusing, and inappropriate to have it intrude on doctor/patient conversations. I think those conversations should be focused on science and medicine, and political statements can be expressed elsewhere. But that’s my personal opinion, as well as that of my husband: he says if his doctor said anything like this, he would find a new doctor. He goes to his doctor for medical reasons, not to be harangued by someone’s political agenda.
We are urged to reform our language, to call ex-cons/felons “Formerly incarcerated/returning citizen/persons with a history of incarceration”. To call diabetics “persons with diabetes”. To avoid the use of the words “blackmail” and “blackball”. To say “enslaved person” rather than “slave.” To avoid using the words “master” and “slave” in software/tech. To say “white” rather than Caucasian, and not to capitalize “White”. To avoid saying target, tackle, combat, or other terms with violent connotation when referring to people, groups, communities, or public health measures (no more “war against drugs” or “war against cancer”.) To replace “homeless” with “people who are experiencing homelessness”.
The AMA says “our current state is built on the land and labor of others in ways that violated the fundamental principles of equity”. And “health equity cannot be achieved without explicit recognition and reconciliation of our country’s twin, fundamental injustices of genocide and forced labor”.
I don’t disagree with that, and I think we should all be more aware of these issues, but I don’t think forcing the discussion into the doctor/patient conversation by specifying the language to be used is the best cure for the inequity that exists throughout our society. Words matter, but our language evolves slowly, not by fiat. The AMA’s Guide is misguided, but it has value in that it will make us think and engage in productive discussions.
        ◦ I have always been skeptical of “politically correct” language, never more so than in the 1970s when I wrote a letter from the flight surgeon’s office to all the missile officers at F.E.Warren Air Force Base. The letter was kicked back to me for revision because it featured “politically incorrect” language. I had used male pronouns, and they said I should have used language that included women. I refused to revise it because women were not allowed to be missile officers for the Minuteman missiles stationed on our base and I knew that every single one of our missile officers was a cis-gender male. They wanted me to include individuals who were not, and could not be, included. “Political correctness” would have been factually wrong. Political correctness has clearly gone too far.

Misogyny in Medicine
Misogyny persists in clinical and academic medicine. We need to be aware of subtle micro aggressions and report abuses, but we mustn’t get carried away into paranoia and conspiracy theories. We should be guided by common sense and moderation in all things.
Harriet Hall on July 13, 2021
Link

In the June 17, 2021 issue of The New England Journal of Medicine Dr. Erica Kaye wrote a “Perspective” piece titled “Misogyny in Medicine”. It made me re-evaluate my own experiences with sexist attitudes. In my book Women Aren’t Supposed to Fly: The Memoirs of a Female Flight Surgeon I described several examples of how I was treated unfairly because I was a woman. When faced with overt discrimination, I was advised by an inspector general not to report it because it would be explained away by the perpetrator and would only harm my career. I never encountered anything approaching overt sexual harassment or assault, but Dr. Kaye opened my eyes to the concept of subtle microaggressions. It made me question a lot of other things that had happened to me during my career and that continue to happen.
Dr. Kaye says that despite increased lip service about the need for gender equity in medicine, misogyny is pervasive in clinical and academic medicine. Women report high rates of discrimination and harassment. According to the National Academies of Science, Engineering, and Medicine (NASEM) at least half of female medical students experience sexual harassment. This “continues into training and faculty careers and correlates significantly with burnout”. Dr. Kaye says that women tend to believe accounts of misogyny at face value, whereas men “interrogate the data, request proof, or seek alternative explanations”. Men tend to see reports of discrimination as outliers, whereas women recognize it as the status quo.
Statistics show that women in medicine make less money and hold fewer leadership positions than men, and women of color bear at least double the burden. Dr. Kaye says when men challenge authority they are described as bold, assertive, or confident; women who do the same are disparaged as aggressive, rebellious, or unprofessional. When she calmly disagreed with a male superior, in response he raised his voice and threatened her job. She realized “that I would be held responsible for ‘provoking’ him and that I needed to absorb his anger and respond with tranquility and equipoise”.
Dr. Kaye gives examples of what she calls microagressions from her superiors:
    • You should take those barrettes out of your hair. You look like a little girl.
    • Your first responsibility is taking care of your husband.
    • My wife stayed home with our kids. It’s too bad your kid is in day care.
    • Your butt looks flatter. Have you lost weight?
She points out that “retaliative microaggressions and discrimination are difficult to prove, readily slipping through policy cracks”. She believes that changing the culture of medicine hinges on the sharing of stories. She encourages women to saturate the mainstream dialog, to flood the discussion with their personal stories of sexism.
As I thought about this, I noticed a potential pitfall. It would be all too easy to blame everything on misogyny. I could look back on every interaction with a male during my career and ask whether it might be an example of subtle microaggressions, whether it might have been different if I were not a female. Paranoia would ensue.
Conclusion: misogyny in medicine is real
Dr. Kaye brings up some excellent points. Yes, misogyny continues in medicine. Yes, incidents need to be reported, and we need to be more aware of what she calls microaggressions. But it would be all too easy to get carried away and engage in paranoia and conspiracy theories.
I just noticed that in the list of editors on the SBM website, my name is listed last, only after the names of two editors emeritus. Did that happen by chance, or should I suspect subtle misogyny? I could complain loudly of mistreatment, but I won’t. I’m happy to give them the benefit of the doubt and say nothing. Let’s not forget common sense and moderation in all things.

The Death of Expertise
In Tom Nichols’ new book, The Death of Expertise, he explains how a misguided intellectual egalitarianism is harming our ability to assess the truth and solve problems, and discusses some of the responsible factors and possible long-term consequences.
Harriet Hall on November 21, 2017
Link

Tom Nichols’ new book The Death of Expertise: The Campaign against Established Knowledge and Why It Matters has direct relevance to many of the issues we are constantly grappling with on Science-Based Medicine. In a democracy, everyone has equal rights. Many people think that means they are equal to experts in knowledge and judgment. In medicine, as in most other areas of public discourse, we are faced with angry laymen who denounce intellectual achievement and scientific knowledge and who distrust experts.
People find ways to reject the evidence when it conflicts with their values and beliefs. When scientific evidence challenges their views, they doubt the science rather than themselves. New examples of this phenomenon can be found every day in the news and in the comments sections of the Science-Based Medicine blog, and trying to set those people straight has proven a mostly futile exercise.
The failure of higher education
Students have become consumers. High school seniors tour college campuses with their parents looking for the one with the best dorms, cafeteria food, and extra-curricular activities, rather than the one that will challenge them and provide the best education. Nichols says colleges are not only failing to provide to their students the basic knowledge and skills that form expertise, they are failing to provide the ability to recognize expertise and to engage productively with experts and other professionals in daily life. They are not being taught “critical thinking: the ability to examine new information and competing ideas dispassionately, logically, and without emotional or personal preconceptions.”
He says students are being treated as clients rather than students. “Many colleges have become hostages to students who demand that their feelings override every other consideration.” Students “explode over imagined slights” and “build about themselves fortresses that no future teacher, expert, or intellectual will ever be able to breach.” They want to be protected from ideas or language they find unpleasant. They are “demanding to run the school while at the same time insisting that they be treated as children.”
The internet
The Internet has provided people with an unprecedented abundance of information, but all too often it gives them the illusion of knowledge, encouraging them to believe they know as much as experts. They hear what they want to hear, and live in a bubble community of people with similar beliefs.
People do not come to the Internet so that their bad information can be corrected or their cherished theories disproven. Rather, they ask the electronic oracle to confirm them in their ignorance.
Nichols says,
…not only is the Internet making many of us dumber, it’s making us meaner: alone behind their keyboards, people argue rather than discuss, and insult rather than listen.
People “power browse” rather than actually reading. We see this all the time on Science-Based Medicine, where commenters criticize an article they obviously have not read carefully or understood. Sometimes I suspect they may just have read the title and seized the opportunity to jump on their particular soap box.
Journalism
The dissemination of “fake news” is an ever more common reality. Most people are very poor at evaluating the reliability of a news source and the truth of what is reported. When a layperson challenges an expert by saying “I read it in the paper” or “I saw it on the news,” it may mean only “I saw something from a source I happen to like and it told me something I wanted to hear.” At that point, discussion has nowhere to go; the real issue is replaced by the effort to untangle which piece of misinformation is driving the conversation. People are constantly barraged with facts and knowledge, but they have become more resistant to facts and knowledge. How did we arrive at this state of affairs? Nichols says, “technology collided with capitalism and gave people what they wanted, even when it wasn’t good for them.”
When the experts are wrong
In our increasingly complex world, we can’t possibly know everything; we have no choice but to trust experts. But sometimes experts get things wrong. Most of the time, their errors are identified and counteracted by other experts. This works so well most of the time that we are shocked when we read about an exception; for instance, when we learn that an incompetent doctor has killed a patient or that a researcher has falsified data. Laymen get exasperated when science “changes its mind,” for instance telling the public eggs are bad for them and then saying no, they’re OK to eat. But that’s not a failure of science, but rather an example of how science works so well in the long run by following the evidence and discarding false provisional conclusions as the evidence improves.
When experts’ errors, fraud, and misconduct are revealed, a layperson naturally asks how we can trust studies in any field. Nichols says that’s the wrong question to ask, because “rarely does a single study make or break a subject.” Single studies are often wrong, but the aggregate of all research is trustworthy. The scientific enterprise as a whole is self-correcting and leads to a consensus of experts that approaches the truth as much as is humanly possible.
The impact on government
Science is essential to rational public policy; it can’t make the decisions, but it provides reality-based information that can guide the decision-makers. Nichols says we have a President who sneers at experts and whose election was “one of the loudest trumpets announcing the impending death of expertise.” He argues that Trump’s campaign was “a one-man campaign against established knowledge.” He provides examples: Trump’s “birther” campaign against Obama, his quoting the National Enquirer approvingly as a source of news. Nichols says rather than being ashamed of his lack of knowledge, Trump exulted in it. “Worse, voters not only didn’t care that Trump is ignorant or wrong, they likely were unable to recognize his ignorance or errors.” He says the Dunning-Kruger effect was at work. It’s not just the things we don’t know (one in five adults think the sun revolves around the Earth), but the smug conviction that we don’t need to know such things in the first place.
He warns,
The relationship between experts and citizens, like almost all relationships in a democracy, is built on trust. When that trust collapses, experts and laypeople become warring factions. And when that happens, democracy itself can enter a death spiral that presents an immediate danger of decay either into rule by the mob or toward elitist technocracy. Both are authoritarian outcomes, and both threaten the United States today.
Conclusion: Hope for the future?
He says Americans no longer understand that democracy only means political equality. They tend to think democracy is a state of actual equality in which everyone’s opinion is as good as everyone else’s, on every subject. Feelings are more important than facts: if people think vaccines are harmful, it is considered “undemocratic” and “elitist” to contradict them.
He sees signs of hope. Experts are rebelling. He cites an angry doctor who asked patients, “Do you remember when you got polio? No, you don’t, because your parents got you [expletive] vaccinated.” He points out that without democracy and secular tolerance, nations have fallen prey to ideological, religious and populist attacks and have suffered terrible fates. But he ends on a hopeful note. He has faith in the American system and hopes that it will eventually establish new ground rules for productive engagement between the educated elite and the society they serve. I hope so too!

The Antithesis of Science-Based Medicine: The Medical Medium’s Fantasy-Based Health Advice
Anthony William, the Medical Medium, hears voices that give him advanced scientific information from the spirit world. He offers reams of health advice based on nothing but fantasy. He even tells readers to call on 12 angels out loud by their name. I call bull.
Harriet Hall on August 1, 2017
Link

Anthony William calls himself the Medical Medium. He claims to have helped tens of thousands of patients heal from ailments that have been misdiagnosed or ineffectively treated. He says he can see into the body like a supercharged MRI scan to diagnose “all blockages, infections, trouble areas, past problems, and even soul fractures.” He hears voices: the Spirit of the Most High (second only to God) talks to him and gives him information that is decades ahead of current scientific knowledge. William disregards science, because science sometimes contradicts itself and is confusing. The information he gets from Spirit is not confusing; it is the truth and the voice of compassion. He has written several best-selling books to share the information that Spirit has given him. And he asks us to believe all this simply because he says so. He has never been tested.
This is all clearly outside the universe of discourse of science-based medicine. We have no reason to take him seriously. I skimmed and dismissed an earlier book of his as not worth my time, but I decided to take a closer look at his book Life-Changing Foods: Save Yourself and the Ones You Love with the Hidden Healing Powers of Fruits and Vegetables. I thought it would make specific claims that would be easy to fact-check. I expected it would be bad, but I was positively gobsmacked by how phenomenally bad it was. I was outraged that he could get away with spreading such egregious nonsense, that a publisher would agree to publish it, and that many readers apparently accept his fairy tales as legitimate health advice (89% 5-star customer reviews on Amazon). It’s a sad demonstration of the fact that common sense is not common, and that critical thinking skills are sorely lacking in our society.
The Holy 4
His Holy 4 are fruits, vegetables, herbs and spices, and wild foods. They are:
the sacred assistance granted to us to make it through life here. These are the answers for humanity. These are what will save us.
Fruits and vegetables are mainstream advice. He makes unsubstantiated health claims for spices. He says wild foods are valuable because they “have survived extremes” and “carry critical survival information in their cells.”
The Holy 4 don’t just contain nutrients:
Because they grow from the earth and are showered by the sun and sky, enduring out in the elements day after day as they form, they are intimately connected to the holy forces of nature. They don’t just contain the building-block nutrients we need to function. They contain intelligence from the Earthly Mother and the heavens that we desperately need about how to adapt.
He provides extensive lists of foods from these four groups. For each, he provides
    • An introduction with unsubstantiated health claims supposedly based on current scientific knowledge (i.e., “Cat’s claw can aid in alleviating almost any symptom, from neurological to digestive.”)
    • A list of conditions that would benefit from bringing the food into your life (for cat’s claw: every type of cancer, Lyme disease, ALS, laryngitis, strep throat, MS, viruses that “are undiscovered,” and many, many more).
    • A list of symptoms that are reasons to bring the food into your life. (For cat’s claw: 36 symptoms ranging from ringing in the ears to Bell’s palsy, from rashes to slurred speech.)
    • Emotional support provided by the food. (Cat’s claw reduces a sense of urgency so you can take time to think rather than automatically reacting to a situation.)
    • Spiritual lesson taught by the food (For instance, “sometimes what we think we seek is within reach.”)
    • Tips on how to use the food (Keep kiwis in a bowl on your nightstand; sleeping beside them will connect them with your being for the greatest life-changing effects. When eating a fig, imagine the fig tree it came from standing in front of you, to amplify its healing, grounding power.)
    • And to top it all off, he offers a recipe featuring each food.
The Unforgiving Four
Radiation, toxic heavy metals, the viral explosion, and DDT are the major culprits. They “ravage our bodies, make us question our own sanity, and push us to the breaking point as a society.”
Life-challenging foods
He makes some false and incoherent claims about these six “life-challenging” foods:
    • Dairy “bogs down the liver.”
    • Eggs feed the viral explosion
    • Corn’s nutritious value has been destroyed by genetic engineering
    • Wheat feeds pathogens
    • Canola oil eats away at all linings in the body: stomach, veins, heart, etc.
    • Food additives labelled as “natural flavors” are really MSG, a neurotoxin that builds up in the brain and destroys nerve cells.
Whaaat?!
Some of his claims are demonstrably false; some are just bizarre. He says 98% of cancers are caused by viruses. He says infertility can be treated by creative visualizations, walking meditations, and breathing exercises to draw white light into the reproductive organs.
He says there are two kinds of living water. Hydrobioactive water is inside all the Holy Four foods. It holds life-giving nutrients and hydrates better than any drink of plain water can. Then there’s the undiscovered cofactor water that contains information to help restore your soul and spirit and to support your emotions.
Angels!?
Just when I thought it couldn’t possibly get any worse, it did. The last chapter is titled “Life-Changing Angels.” He really believes that angels exist. Twelve of them, and he lists their names, like the Angel of Abundance and the Angel of Addiction. He instructs his readers to call on the appropriate angel out loud, by name. For instance, “Angel of Provision, please help me.”
Celebrities praise him
Gwyneth Paltrow (ugh):
While there is most definitely an element of otherworldly mystery to the work he does, much of what Anthony William shines a spotlight on—particularly around autoimmune disease—feels inherently right and true. What’s better is that the protocols he recommends are natural, accessible, and easy to do.
In the Foreword to the book, Christiane Northrup, MD (ugh!), says William has “a gift that has made him a conduit for information that is light-years ahead of where science is today.” She says wild blueberries have survived in harsh environments against all odds, so eating them brings that tenacity into your body. They are a power food, a “food of resurrection.” She agrees that each fruit or vegetable is imbued with the specific emotional and spiritual support it provides when you eat it. She believes life-changing angels fortify our food supply; they “encourage every apple and lettuce leaf and everything else to grow, help bring food to the hungry, work to disarm GMO food production, assist with and support the organic food movement, and even influence weather patterns.”
Scientists don’t
Skeptical Raptor is not impressed. His extensive and well-reasoned critique concludes:
The Medical Medium uses the pseudoscience of spiritual readings plus the pseudoscience of alternative medicine to diagnose and treat patients. Be aware. Be very aware.
Few of William’s ideas are original. He regurgitates alternative medicine claims for dietary supplements, and recommends “detoxification” and even homeopathy. Vitalism and magical thinking permeate his writing.
Who wants health advice guided by fantasy?
I am appalled by the sheer volume of claims in this book, especially the long lists of conditions and symptoms associated with each food. William’s belief system has no grounding in reality or science. Vitalism? Magical thinking? Talking to angels? Getting advanced knowledge from Spirit? Two kinds of water? How can any reasonably intelligent, educated person believe this nonsense? Accepting the Medical Medium’s assertions requires a degree of gullibility I can’t even begin to imagine. Especially for an MD who has studied medicine and science. Christiane, what happened to you?
Anthony William is a throwback to a primitive, prescientific age. Our ancestors listened to shamans and witch doctors; we should know better today. There is no reason to think the voices William hears are divine guidance; they are hallucinations until proven otherwise. His advice is not science-based, evidence-based, or reality-based; it is fantasy-based. He does emphasize eating more fruits and vegetables, which is a good thing – which is why doctors recommend it. He may get a few things right by chance, but it’s impossible to tease out those few things from the masses of chaff that engulf them; and much of what he says is demonstrably wrong.
Either Anthony William is a deliberate charlatan cynically making stuff up for profit, or he is a self-deluded believer who interprets his inner voices as real communications from the spirit world and is reinforced by cold reading and confirmation bias. Either way, it would be foolish to rely on him for health advice.

Patient Groups and Pseudoscience
Steven Novella on August 24, 2016
Link


The biggest challenge we face promoting high standards of science in medicine is not making our case to the community. Our case is rock solid, in my opinion, and backed by evidence and logic. There is no question, for example, that homeopathy is 100% bogus and should not be part of modern medicine.
Our challenge is that there are literally billions of dollars to be made selling fake medicine and dubious treatments. This means that unscientific practitioners have an immediate financial incentive to promote themselves and their treatments, and they will tirelessly do so, on any front they can find. Further, the stars of unscientific medicine have the resources to do so – to intimidate critics, cozy up to politicians, open centers in respected hospitals, and market their brand.
We simply don’t have the manpower to confront them on every front, and the mainstream scientific and medical communities are frankly just not paying enough attention. They are largely unaware that pseudoscience is infiltrating their profession right under their noses, or they have been lulled into thinking this is a small and benign phenomenon.
Patient groups
These many fronts in which science confronts pseudoscience include the media, hospitals, continuing education, journals, the marketplace, politics and regulation, and research funding.
I cannot remember if we have ever written about another front that does not get much attention – patient groups. For just about every disease there is a grass-roots group of patients, their families, and experts and others interested in the disease. These groups raise money for research and to help patients with services, promote awareness of their disease, run support groups, and act as a source for information and referrals.
Typically each group is independent and is run by dedicate lay people. They provide an incredible service and are invaluable.
They are also a convenient target for charlatans, and yet another front on which we need to confront pseudoscience. For example, I recently received the following e-mail:
I just wanted to thank you for the work you do re pseudoscience. I have been reading your blog on Science Based Medicine regarding alternative medicine. I am the parent of a grown child with severe bipolar disorder (he’s progressed well on medicine and therapy) so we belong to some support groups. Problem is they want to have lecture series on alternative medicine, including naturopathy. Recently I’ve been educating myself on the subject, to the point where I am extremely concerned by what I’m reading. My concerns have been dismissed by these groups–International Bipolar Association and DBSA. Earlier in my life I was familiar with the works of James Randi and others, so I already had some basic understanding of the educational work you do. There’s much much more I would like to say, but that will have to wait. Again, thank you thank you.
I have received many such e-mails over the years, and have encountered this phenomenon directly myself. CAM proponents have found they can market themselves and their treatments directly to patients by giving “educational lectures” to patient groups. These lectures are just sales pitches. I have even encountered this in public schools, offering “health lectures” promoting things like acupuncture or chiropractic.
Patient groups are an ideal target for such pseudoscientific promotion. They are typically run by lay people who have educated themselves about their disease but lack formal training in science. They are clearly passionate about supporting patients, which often means supplementing standard care with other resources.
When a naturopath or chiropractor comes calling, offering those extra resources their doctors may not be offering, that seems to fit right in with their mission. Once a few key people in a patient group are indoctrinated into the CAM philosophy, the patient group basically becomes a marketing organ for pseudoscience.
If some members, like the e-mailer above, are uncomfortable with the pseudoscience, they generally lack the expertise or confidence to effectively oppose it. Outside experts have been preemptively portrayed as closed-minded or protective of their turf.
This phenomenon creates a thousand new fronts on which to confront medical pseudoscience.
What to do?
There is no easy solution to this problem. SBM tries to be a resource for those people, like the e-mailer, who recognize the dangers of pseudoscience. It gives them the specific information and the confidence to more effectively hold the line, but clearly it is not enough to just provide an information resource.
We do not have enough experts in science-based medicine and pseudoscience to effective confront pseudoscience in each patient group. What would need to happen is for the doctors and scientists who are experts in each specific disease to vigorously take on the role of promoting proper standards of science in such groups. In fact, they need to do it proactively – bake such standards, and awareness of health fraud and dubious treatments, into the resources of the group itself. I have seen this done effectively as well.
Further, patient advocates need to understand the nature of pseudoscience in medicine and that if they truly wish to advocate for their patients they need to fight against those who wish to exploit them.
The ultimate solution, however, is to have proper regulation of medicine in the first place.

Cargo Cult Psychology
Harriet Hall on March 15, 2016
Link


Last year I reviewed Tomasz Witkowski and Maciej Zatonski’s book Psychology Gone Wrong where they pointed out that many of psychology’s accepted beliefs and therapies were not based on good evidence. Now Witkowski has written a new book, to be published later this year, Psychology Led Astray: Cargo Cult in Science and Therapy, that is certain to ruffle a lot of feathers. He compares psychology to cargo cults. He says, “the cargo cult phenomenon is virtually absent in physics, chemistry, and mathematics, yet it runs rampant in both psychology and sociology.” He uncovers cargo cult practices in psychology, unmasking therapies that are devoid of science, dangerous, and even cruel, especially those directed at children.
Richard Feynman was the first to compare the contemporary social sciences, including psychology, to a cargo cult. For those not familiar, the term “cargo cult” originates with the natives in Melanesia, who were awestruck by the planes that landed on their islands during WWII bringing all kinds of supplies. They had no understanding of what airplanes were or where they came from. Magical thinking led them to create bamboo replicas of planes and control towers in the superstitious belief that it would attract planes and bring them material goods. Similarly, many psychology researchers have been imitating the methods of science without really understanding how science is supposed to work. They go through the motions, but their research designs are so poorly thought out and the methodology so poor that their results are meaningless. And then they use those meaningless results to guide therapy. They have been led astray, have deceived themselves, and have harmed patients.
Witkowski’s book starts with a brilliant discussion of how the human mind works and how it evolved thinking processes that frequently lead to errors. Next he provides a thorough discussion of cargo cults, with far more detail than I have read elsewhere. It’s fascinating stuff, especially about the huge variety of cargo cults (at least 55) and the ones that have persisted into the 21st century because of deliberate deception, self-deception, and confusion of reality and beliefs.
Is psychology a science or a deception?
If psychologists were good scientists and psychotherapies based on evidence were effective, we would expect psychology to have had more of an impact on the world by now. We would expect to see an improvement in statistics regarding mental health and crime, fewer people suffering from depression, fewer suicides, etc. In reality we find that as the number of psychotherapists rises, the number of patients rises, and the mental health statistics get worse.
John Holt did not have much doubt in this regard:
The person whose main lifework is helping others needs and must have others who need his help. The helper feeds and thrives on helplessness, creates the helplessness he needs.
Witkowski asks,
Are we involved in an act of profound deception where we bear responsibility for the upkeep of thousands of scientists who engage themselves in utterly idle efforts?
500 psychotherapies
There are 500 kinds of psychotherapy; few disappear. The most successful are integrated with popular beliefs and traditions. Cargo cult prophets discourage empirical validation. Freud said “the wealth of reliable observations on which these assertions rest makes them independent of experimental verification.” Witkowski explains why people believe in ineffective treatments and why “American clinical psychologists put more trust in their own clinical experience supported by reports of their colleagues than in the scientific evidence”
To recommend a method whose consequences include the systematic and daily experience of suffering, one must possess an overwhelming certainty that the positive effects of the therapy are sufficient to counterbalance its negative ones. Unfortunately, pseudoscientific psychologists do! They have strong convictions just like the majority of cargo cult priests do. Convictions built without a foundation, resting on the pillars of mere opinion, enhanced with thin anecdotal evidence; there is precious little empirical proof to be found.
Why is this so common? He covers a long list of factors that contribute to the blossoming of psychobusiness. Anything can be called therapy today. We have color therapy, sand play therapy, nature walk therapy, and laughing therapy. If parents buy a dog, does that mean their child is getting dog therapy?
He provides numerous examples of prominent cargo cult therapies that have become well established and are rarely questioned. He suggests they should become the subject of investigations by ethnographers whose field of interest encompasses analysis of cults, myths, rites, superstitions, and other man-made constructs.
Adult Children of Alcoholics
Adult children of alcoholics (ACoA) is a recently coined diagnostic category with a laundry list of symptoms that are used to support the diagnosis. The concept has been expanded to include dysfunctional families, including most of the population.
The approach acclaimed by the ACoA movement and affiliated therapists is a peculiar brew of common sense, half-truths and common observations on psychology, metaphors, false assertions and harmful recommendations.
People waste their lives exploring imaginary problems or things that will not necessarily lead to solving their real problems. Harms include labels and social stigma, belief that we are not responsible, and digging up dirt from the past.
Psychotherapy for cancer?
The Simonton “mind over matter” method of psycho-oncology is alleged to double survival rates in cancer patients. It’s basically a marketing lie based on faulty research. Contrary to common belief, there is no association between emotions, stress, and cancer rates. Scientists have failed to identify any link between positive attitudes, emotional states, and cancer survival. In some properly-done studies we can even observe the opposite effect; workplace stress is associated with a reduced risk of breast cancer! Encouraging patients to think positively is not only an extra burden imposed on them in a difficult situation, but also deprives them of vital social support.
As the American Cancer Society concluded:
After careful study of the literature and other information available to it, the American Cancer Society does not have evidence that treatment with O. Carl Simonton’s psychotherapy method results in objective benefit in the treatment of cancer in human beings.
Cognitive-behavioral therapy, encompassing training in relaxation, stress-management and problem-solving techniques, has been shown to produce better mood, reduce psychological suffering, and improve perception of quality of life in cancer patients. But it doesn’t prolong their survival.
The disaster industry
Critical incident psychological debriefing is another example of cargo cult psychology. A Cochrane review concluded:
There is no current evidence that psychological debriefing is a useful treatment of posttraumatic stress disorder after traumatic incidents. Compulsory debriefing of victims of trauma should cease.
The helpers felt better. Sometimes the patients felt worse.
The current WHO recommendations on psychological debriefing say:
Psychological debriefing should not be used for people exposed recently to a traumatic event as an intervention to reduce the risk of posttraumatic stress, anxiety or depressive symptoms. Strength of recommendation: STRONG.
Child abuse and related nonsense
The Doman-Delacato Method is intended to treat brain damage in children and to increase the intellectual potential of non-impaired children. They claim that
our individual genetic potential is that of Leonardo, Shakespeare, Mozart, Michelangelo, Edison, and Einstein. … Our individual potential race is not that of our parents or grandparents. … All intelligence is a product of the environment.
They make fantastic claims like having taught one-year-old old brain injured children to read. They use ridiculous treatments requiring a tremendous time investment, with patterning, masking, hanging the child upside down, vitamins, diet, etc. They have built a worldwide empire. Witkowski’s critique is devastating.
Educational kinesiology (Paul Dennison and Brain Gym®) has been funded by government and is used in 87 countries. Ben Goldacre calls it “a scientific explanatory framework that is barkingly out to lunch.”
Attachment therapy and holding therapy has led to child torture and numerous deaths.
Studies use unreliable measures of problems and treatment results. They rely on subjective reports from parents and on questionnaires that have not been validated. These methods continue to be used because of unprincipled profit-motivated people, ignorance, or inability to admit they were wrong.
The list goes on
There is no end to psychological pseudoscience and pseudotherapies. A few of the other topics Witkowski covers (some covered here on SBM):
    • Facilitated communication
    • Neurolinguistic programming
    • Dolphin assisted therapy
    • The Vojta method of rehabilitation by “priming”
    • The Tomatis method of auditory integration
    • Sensory integration therapy
    • Imaging as the new phrenology
    • Dr. Amen’s SPECT scans
    • Prefrontal lobotomies (an all-too-rare example of a cargo cult therapy that was finally rejected).
A suggested remedy
Baker and his colleagues have recommended changes in the existing accreditation system within the APA. They would stigmatize unscientific training programs, as well as those practitioners who use methods that are scientifically unproven. They justify this call with similar changes that have been successfully put into practice in medicine.
They say all therapies that are not empirically confirmed should be classified as an “experimental therapy,” and patients should be recognized as “medical subjects.” They suggested that using problematic therapeutic interventions such as the “facilitated communication method” in the case of autistic children should be treated as a criminal act.
I would propose the term “therapies in the phase of uncontrolled experiments on humans” as an alternative.
Guidelines for patients
When deciding to undergo therapy, one should remember that we are choosing from at least a few hundred schools and modalities, many of which are in conflict with one another. Only a handful of them can boast empirically proven effectiveness.
Patients should ask these questions:
    • Is there evidence for the treatment?
    • Is there a way to get redress if the treatment causes harm?
    • Is the therapist a member of a professional society with a code of ethics or a code of good practices?
    • Is there a system of supervision of therapists?
    • What will happen if you lose the ability to pay or if you move away?
    • Does your therapist record the course of therapy in a manner enabling him to transfer you as a patient to another specialist?
    • Does your therapist follow precise procedures and standards of operation so that the new therapist will be able to determine what has been done and what still needs to be addressed?
    • Did your therapist define the purpose of the therapy, and is he able to determine when the goals will be achieved? Or, rather, does he treat the whole process as a creative act, in which you are the medium, and nobody apart from him is able to “complete the masterpiece”?”
    • Does your therapist have liability insurance?
    • Are your therapist’s belief systems compatible with your own – atheist, Catholic?
Conclusion: It’s not all doom and gloom

It’s not all doom and gloom. The book ends on a positive note, with a letter to Richard Feynman chastising him for calling the whole field a cargo cult. Witkowsi draws Feynman’s attention to examples of good science in psychology that have had demonstrable benefits for society, for instance the studies on reaction time in response to different configurations of brake lights that led to laws requiring a third brake light on vehicles, dramatically reducing the number of rear-end collisions, injuries, and repair costs.
There are some errors in the book: AIDS is lumped with untreatable diseases, and the chapter on statistics is poor, with a misleading explanation of p values. But on the whole, there is a lot of valuable information here. Even if you don’t agree with calling these travesties of science cargo cults, it will make you wonder which other generally-accepted psychological principles and therapies are based on good science. This book is the second part of a trilogy; I look forward to volume 3.

Communicating Health Science News
Steven Novella on December 17, 2014
Link


A recent study addresses the problem of sensationalism in the communication of science news, an issue we deal with on a regular basis. The study was titled “The association between exaggeration in health related science news and academic press releases: retrospective observational study“. The results show two interesting things – that university press releases frequently overhype the results of studies, and that this has a dramatic effect on overall reporting about the research.
The authors reviewed “Press releases (n=462) on biomedical and health related science issued by 20 leading UK universities in 2011, alongside their associated peer reviewed research papers and news stories (n=668).” They found that 40% of the press releases contained exaggerated health advice, 33% overemphasized the causal connection, and 36% exaggerated the ability to extrapolate animal and cell data to humans.”
Further:
When press releases contained such exaggeration, 58%, 81%, and 86% of news stories, respectively, contained similar exaggeration, compared with exaggeration rates of 17%, 18%, and 10% in news when the press releases were not exaggerated.
This study points a finger directly at academic press offices as a significant source of bad science news reporting. This does not let other links in the news chain off the hook, however.
The problem is worsened by changes over the last decade in the news infrastructure. The internet and changing business models make it difficult for large news outlets to maintain specialist journalists and editors. Therefore science and health news is more frequently being reported by generalist reporters, and not filtered through a dedicated science editor.
As a result, some outlets simply reprint science press releases without doing any independent investigation, without talking to experts who did not author the study, and without any ability to put the new research into a proper context. This is evident simply by searching on a science news item and finding dozens of websites publishing the exact same copy, word for word, which ultimately leads back to the press release.
Bad science news reporting that I have covered over the years has demonstrated problems at every level, sometimes even starting with the scientists themselves. In some cases the conclusion or discussion of a published study will contain speculation that goes well beyond the data. The discussion is a legitimate place to explore the implications of a study, but authors need to be clear about what conclusions flow from the study and what questions are prompted by the study but not addressed by its data.
Authors also should avoid the temptation to draw causal conclusions from correlations and associations. In fact, they need to do more than avoid making spurious conclusions themselves. The discussion is a great place to anticipate and discuss what the study cannot be used to conclude. It is expected that authors will discuss the limitations of a study, but often they can be more explicit and anticipate common abuses.
The BMJ study above discusses two very common ways in which studies are overhyped. The first is drawing causal conclusion from correlations. Correlations alone do not prove causation, but a pattern of correlation can suggest some causal relationships over others. This can often be a complex and nuanced evaluation, and study authors should do their best to put correlations into a proper context.
Another potential source of hype is extrapolating from one type of data to conclusions that the data does not directly address. In healthcare reporting the most common form of this problem is extrapolating from preclinical data to clinical conclusions. Preclinical data includes in-vitro studies looking at cell cultures, or animal data. Such evidence should only be used as a means of determining whether or not human clinical trials are likely to be safe and useful.
Also, there are different types of clinical studies, and often they are misrepresented. For example, pragmatic studies are used to compare the use of treatments which have already been established as effective in real-world clinical settings. They are not designed to test efficacy, because they are typically not blinded placebo comparisons. Yet it is not uncommon to use unblinded pragmatic studies to make efficacy claims.
The biggest problem, however, is reporting preliminary or exploratory data as if it is confirmatory. I have argued previously that preliminary data should not even be reported to the press, in most cases. Such data simply does not rise to the level of newsworthiness, as it is far more likely to be wrong than to pan out in later research. If preliminary data is communicated, it should come with a bold disclaimer about the preliminary nature of the study.
Sometimes the scientists themselves are innocent, but their university press office tries to find an angle that will promote the study by making it seem relevant to some headline-grabbing issue. Their job is to get press attention, and dry pre-clinical studies are unlikely to do the job. Therefore they turn to a number of common ploys to make any research seem relevant to a topic that is likely to grab media attention.
In the health arena, this means any study involving viruses might cure the common cold, any study involving cell metabolism might lead to a cure for cancer or obesity, and any study of brain cells might cure Alzheimer’s disease.
Sometimes a tiny footnote in the discussion is latched onto and presented as if it were the main focus of the study, even when the study actually has nothing to do with the topic.
Scientists are usually given the opportunity to review the university press release before it is made public. This means that scientists have an opportunity to be more active in making sure that their research is presented properly.
Of course, journalists and news outlets have responsibility for the news they report, and should not simply be reprinting press releases word-for-word. Science journalists need to at least read the original study and see if the results match what is being sold in the press release. Ideally they will also talk to independent experts to put the study into context.
Conclusion
The BMJ study is not surprising and fits in with my experience reporting science and health news over the last decade. I was a bit surprised at how much an effect the press releases were having on later reporting. This is actually good news, because it means that if scientists and university press offices were more responsible, then a significant amount of bad science news reporting can be avoided.
Scientists are press offices should follow some basic rules for good science news reporting:
    • Put a study into an overall scientific context
    • Do not emphasize minor aspects of the research simply because they are more exciting
    • Do not search for any health implication of pre-clinical data just for the headline, and certainly don’t make it seem like the focus of the study
    • Anticipate and explicitly address common misinterpretations of the research
    • Consider not reporting preliminary data at all, or at least clearly label preliminary data as such early in the press release and make is absolutely clear what this means
    • Do not report correlations as if they prove a specific causation
    • Be up front about all the limitations of a study and alternate interpretations
    • Make it clear when a study is an outlier, if there are multiple schools of thought, or where the study lies in relation to the current preponderance of expert opinion; in other words, do not present one small study as if it overturns a well-established consensus
In short, a good press release not only summarizes the results and conclusions of a study, but educates anyone reading it (journalists and the public) about the kind of research being done and how it fits into the overall scientific enterprise.
In my opinion, this falls into the broader mission of universities, which is education. They should not only be concerned about the education of their students, but also the public at large. A great deal of the education of the public about scientific matters (or other academic matters) is handled out of the university press office. Greater attention needs to be paid to their role in communicating science.

How to Think
Harriet Hall on February 11, 2014
Link
Robert Todd Carroll, the author of The Skeptic’s Dictionary, has a new book out: The Critical Thinker’s Dictionary: Biases, Fallacies, and Illusion and what you can do about them. Since some of our commenters and most of the CAM advocates we critique are constantly committing logical fallacies, a survey of logical fallacies is a good idea both for us and for them, and this book fits the bill.
When I received the book in the mail, I set it aside, thinking it would be a somewhat boring listing of things I already knew. When I finally got around to reading it, I was surprised and delighted. It held my interest, reminded me of things I had forgotten, explained other things I had never heard of, and provided entertaining stories to illustrate each point. Best of all, the bulk of his examples are taken from medicine and relate directly to the topics we discuss on SBM.
Carroll is well-qualified to write about logical fallacies: he is a retired professor of philosophy who has long promoted skepticism and taught classes in critical thinking, and he writes in an entertaining, accessible style. He started The Skeptic’s Dictionary website in 1994 with 50 articles and it has now grown to several hundred articles. It attracts more than a million visitors a month, and some of its entries have been translated into more than a dozen languages. It has become a go-to reference for anyone seeking the facts on questionable claims about everything from crop circles to homeopathy. Its articles are thorough and well documented with lots of references and links.
I am partly responsible for this book, as he explains in the acknowledgements and Preface. I reviewed his e-book Unnatural Acts: Critical Thinking, Skepticism, and Science Exposed! for Skeptical Inquirer in 2012. The original version of that book included a list of 59 fallacies, biases, and illusions that readers could learn about on their own. I suggested that he add a short description of each fallacy rather than simply listing their names, and he did. This led him to expand on the descriptive paragraphs in a blog, addressing one each week for 59 weeks. The new book is a re-organized and improved version of those blog posts.
He starts with a case study of an anti-vaccine activist, Stephanie Messenger, who wrote the unfortunate Melanie’s Marvelous Measles, a book that teaches unvaccinated children to welcome childhood diseases. He tells her story with insight and understanding. She had a child who became ill after vaccination and eventually died. She was desperate to understand why and to have the power to protect her other children from a similar fate. The specialists caring for her child suspected Alexander disease, a rare genetic neurodegenerative disease, but for some reason she rejected that possibility and became convinced that vaccines were responsible. Carroll details the series of fallacies and illusions that misled her and reinforced her convictions: the post hoc ergo propter hoc fallacy, confirmation bias, community reinforcement, the reverse halo effect, the illusion of understanding, and the illusion of control. He makes it very easy to understand why she thinks as she does. Hers was a very natural human response, but with education in critical thinking, we can hope to overcome those natural tendencies and avoid the kind of errors she made.
In a discussion of the ad hominem fallacy, he says something that I wish our critics would read and take to heart before assaulting us in our comments threads:
To refute my argument, you must show that my evidence is insufficient, that it is based on false or questionable assumptions, that the evidence I present is irrelevant, that I’ve omitted important evidence, or that I’ve given improper weight to various piece of evidence.
It’s not enough to accuse us of bias, financial gain, or pimping for Big Pharma; you have to address what we wrote rather than who we are, and show that what we have written is wrong (with credible references, please!).
His description of how his article on EMDR (eye movement desensitization and reprocessing) was attacked is typical of what we have repeatedly seen here on SBM. His critic:
    • asked if all his books were critical of something he knew nothing about
    • said EMDR had worked for her (anecdotal)
    • explained that there were exercises to do between sessions (which actually confirmed Carroll’s point that EMDR was essentially cognitive behavioral therapy)
    • complained that he should have taken EMDR training before he wrote
    • argued that he couldn’t possibly understand because he hadn’t experienced the agony of PTSD for himself
    • called him names
    • said his logic skills were atrocious and wondered how he ever got a professorship
    • said he doesn’t benefit anyone by saying EMDR doesn’t work (he didn’t claim that it doesn’t work, he claimed that it was a deceptive way of packaging cognitive behavioral therapy; she was attacking a straw man position that he did not hold)
He uses the NCCAM (National Center for Complementary and Alternative Medicine claim that 38% of Americans use alternative medicine to illustrate the ad populum fallacy. He illustrates other fallacies with examples from medicine. He discusses fears about cellphones and vaccines, testimonials, placebo effects, the appeal to tradition (“acupuncture is thousands of years old”), the appeal to authority (“Dr. Jay Gordon says vaccines aren’t safe.”) and how “authority” is not the same as expert scientific consensus.
He illustrates the availability heuristic with an experience Dr. Jerome Groopman had. He misdiagnosed a patient suffering from aspirin toxicity because her symptoms were compatible with the many cases of viral pneumonia he had recently been swamped with, so that the viral pneumonia diagnosis was readily available to his mind while aspirin toxicity was not.
In begging the question, the conclusion of the argument is entailed in its premises: abortion is murder, murder is illegal, so abortion should be illegal.
He covers a lot of recent psychological research into phenomena like change blindness, priming, the backfire effect, and many, many others.
He discusses the illusion of skill and the illusion of understanding (We can relate to that: we’ve seen many examples of the Dunning-Kruger effect among our commenters).
The clustering illusion: several cases of cancer in a neighborhood can occur by chance and don’t necessarily mean the inhabitants are being exposed to carcinogens in the soil or water. Communal reinforcement is when a claim becomes a strong belief through repeated assertion by members of a community. Confabulation: we all do it; our memories sometimes mix fact and fiction to recall things that never happened, and on a daily basis we invent reasons to explain how we feel.
Selection bias: Edzard Ernst tells how, as a young doctor, he gave mistletoe injections to cancer patients and was impressed by the results until he realized that his hospital was known throughout Germany for its approach, and desperate patients went there because they wanted that type of treatment and had high expectations.
Under causal fallacies (correlation is not causation) he covers the problem of multiple endpoints in small randomized controlled trials, where spurious correlations are expected due to the laws of chance. He also tells how doctors were interpreting signs of disc disease on MRI scans as an indication for back surgery until someone thought to do MRIs on patients without back pain and found that 2/3 of those showed the same changes.
I already knew the word pareidolia, but apophenia was new to me. It means the spontaneous perception of connections and meaningfulness of unrelated phenomena, the tendency to find personal information in noise, seeing patterns where there are none, the kind of subjective validation that cold reading exploits.
This is only a brief taste of the profusion of riches in this book. I highly recommend it to everyone. Whether you are a novice who needs a primer to learn about detecting logical fallacies or are already a seasoned pro, you are certain to learn something from The Critical Thinker’s Dictionary, and you are sure to enjoy the experience.

Progressive Mythology
Harriet Hall on May 21, 2013
Link
In their book Science Left Behind: Feel-Good Fallacies and the Rise of the Anti-Scientific Left, Alex Berezow and Hank Campbell counter allegations of a Republican war on science by pointing out how political progressives are equally anti-science. According to Berezow and Campbell, progressives hold opinions that are not based on physical reality, and claim that their beliefs are based on science even when they are not.
I try to stay out of politics, but anti-science attitudes should be discouraged wherever they are found, and the mythology of progressives as described by Berezow and Campbell is very much like the thinking of alternative medicine:
     1.  Everything natural is good
     2.  Everything unnatural is bad
     3.  Unchecked science and progress will destroy us
     4.  Science is only relative anyway
I wasn’t clear on what “progressives” meant, but apparently progressives are similar to liberals in that they value economic authoritarianism and different in that they are also social authoritarians. Liberal economic authoritarianism favors higher taxes on the wealthy, more regulations on the marketplace, and social programs that redirect money to target social inequality. Conservatives want to limit such government interference with the economy; but they are social authoritarians like progressives, only on different issues. Where conservative social authoritarians want government to ban “immoral” things like sex and drugs, progressive social authoritarians endorse government control over the environment, food production, and education. Conservatives want to ban abortion; progressives want to ban plastic grocery bags. Here’s another definition:
…”liberals” in our current parlance are those who focus on using taxpayer money to help better society. A “progressive” are [sic] those who focus on using government power to make large institutions play by a set of rules…A liberal policy towards prescription drugs is one that would throw a lot of taxpayer cash at the pharmaceutical industry to get them to provide medicine to the poor; A progressive prescription drug policy would be one that centered around price regulations…
Republicans have been criticized for their anti-science stance on evolution, global warming, and stem cell research. There is an equally disturbing tendency for activists on the other side of the aisle to cherry pick, misinterpret, misrepresent, and abuse science to advance their ideological and political agendas. They have misused science to attack vaccines and genetically modified foods, to promote organic food, and to propose poorly thought out environmental protection legislation. When science is co-opted to serve ideology, science is degraded and the resulting public policies do more harm than good.
The authors present many examples of progressive ideology’s misuse of science and support of injudicious policies, for example:
    • One would think animal rights activists, conservationists, and food fetishists would all be enthusiastic about innovations to improve the future of food production, like laboratory-grown meat and technology to improve agricultural efficiency; but they typically reject them, perhaps because they think of them as “unnatural.”
    • “It takes 1 gallon of gas to make 1 pound of beef” is a false claim, as is the claim that walking to the store creates 4 times as much emissions as driving to the store. These claims are based on math and reasoning errors: they took a fact out of context, mixed terms, and guessed.
    • Mandatory low flush toilets have inconvenienced us all, and they have also caused sludge accumulation and odors so bad that San Francisco is spending $14 million to dump bleach into the sewers to combat it (and bleach is not exactly environmentally friendly). Domestic water use only represents 1% of total use and toilets are a small fraction of that; it would make far more sense to target efficiency in power plants (49% of water use) and irrigation (31%). The toilets were a typical “Band-Aid” fix that produced “feel-good” benefits for environmentalists but essentially accomplished nothing for the environment. Ironically, many of the same people who advocate the right to choose for vaccines were quite happy to legislate away our right to choose for toilets.
    • They lament the birds killed by wind turbines, which amount to only 0.006 percent of total bird fatalities; domestic cats kill about 500 times as many birds. Surely it would make more sense to reduce cat numbers than wind farm numbers (or to encourage cat owners to keep their pets indoors, which is healthier for the cat and increases its life expectancy by several years). But cats are “natural” and wind farms are not.
    • They worry about possible harm to caribou and to native cultures from oil development in Alaska. But the caribou population is nine times larger in Prudhoe Bay since oil was discovered there, and the local natives who depend on caribou strongly support responsible development.
    • Even Al Gore has admitted that corn for fuel was a mistake based on faulty science: it was worse for the environment and raised prices for poor people who needed corn for food.
Pragmatism is notably absent from progressive thinking. Environmentalists tend to be inflexible absolutists, unwilling to balance the trade-offs between protecting the environment and promoting economic development. They tend not to consider economic realities. They push the precautionary principle to an extreme without considering the costs of alternatives and the risks of adverse unintended consequences.
The chapter on “vaccines and Viagra” is pure music to the ears of science-based medicine. They agree that the anti-vaccine movement is based on outright lies, they call the Huffington Post a laughingstock of the scientific community for its endorsement of CAM, they call for the NCCAM to be abolished, they explain why presenting data about relative risks rather than absolute risks is misleading, and they point out that:
Just because a published paper presents a statistically significant result does not mean it necessarily has a biologically meaningful effect.
Scientifically studying gender or racial differences is discouraged, if not completely taboo. It would be politically incorrect to find evidence suggesting that abilities are not fairly distributed. A double standard is at work. The smaller percentage of women in science and math is seen as a problem, but the much smaller percentage of men in the social sciences is not. When social psychologists see women or minorities under-represented, they see unfairness and discrimination; but conservatives are under-represented among social psychologists by a factor of more than 100 and no one seems to be worried about that. Academic science has gradually become more representative in race and gender, but has become far less representative in politics: it has become dominated by progressives.
Progressive ideology has a simple fix for complex problems: more legislation. The progressive European Union strives to regulate even the irrelevant minutiae of its citizens’ lives. Science and common sense are casualties of what amounts to a bureaucratic war on reason.
The usual division of political views into two groups (liberal vs. conservative) or four (liberal, conservative, libertarian, and progressive) doesn’t work well. Individuals are more nuanced and don’t fit into well-defined pigeonholes. Berezow and Campbell propose another schema based on values: a triangle with freedom, fairness, and excellence at the corners. Science would fall close to the excellence point of the triangle and more on the freedom side. Fairness is less important to science: it would be silly to treat all scientists as if they were equally competent or to consider all research projects equally deserving of funding. Science must be a meritocracy.
The US leads the world in science, has the most Nobel Prize winners and more of the top 100 universities than any other country or region. Progressive European countries lag far behind. Even if we do worse than other countries on science and math tests, we seem to be better at creativity and independent thinking, and at getting results. We need to figure out how to best support those qualities rather than just throwing money at schools and trying to raise test scores.
Berezow and Campbell call for clear, unbiased thinking about public policies based on good scientific evidence rather than ideology-influenced distortions of science. They call for more basic research, more transparency, better communication of science to the public, more scientific rigor and pursuit of excellence. They offer practical proposals for improvement. Above all, they demand that we quit politicizing science. How could anyone disagree with that?
You may very well disagree with some of the opinions in this book, or even with the way it selects its facts, but it will give you food for thought about some very important issues. I recommend it. Especially if you have read Chris Mooney’s book The Republican War on Science.

The Misinformation Wars
Steven Novella on April 9, 2025
Link
The core mission of SBM comes down to a few things – examining the complex relationship between scientific evidence and healthcare, exploring issues of how optimally to regulate health care and health products, and fighting misinformation. Over the years I think we have made some solid strides on the first category. The medical infrastructure has been trending increasing towards higher standards of scientific evidence and greater awareness of the many pitfalls of bad science or pseudoscience. More of my colleagues know what p-hacking is, and the higher quality journals are exploring ways to improve the quality of scientific evidence.
The other two broad areas have not been trending well. Regulations remain a mixed bag and overall have been ratcheting in the wrong direction. The forces of quackery are relentless at promoting their interests, and have the ill-gotten gains to back their efforts. These efforts include watering down science-based standards, carving out exceptions for specific pseudosciences, and legitimizing pseudoscientific practitioners, practices, and products.
Regarding misinformation, it’s hard not to feel as if we are massively losing the battle. I do think the situation would be far worse were it not for the efforts of fact-checkers and analysts trying to set the record straight, but at least over the medium term misinformation is clearly on the rise. A lot of this is likely due to social media and algorithms optimized for engagement rather than quality of information. Also, the purveyors of misinformation are just getting better at what they do, and have been building infrastructure to extensively spread their misinformation and disinformation and work it into the mainstream conversation.
I think they have also figured out what we have known for a long time at SBM (and the broader skeptical community) – that misinformation is the endgame. Unfortunately, mainstream institutions have not figured this out, although they might be slowly waking up to this reality. It doesn’t matter if your claims or practice are backed by super high quality science, or if you have thoughtful and effective regulations, if the public believes abject nonsense. Carl Sagan warned about this in his Cosmos series in the 1980s – if the public does not understand and support science, eventually the barbarians will come and tear it all down. I think that is what we are experiencing right now.
Therefore, understanding misinformation and how best to limit and counter it is critical. Along those lines, a recent study looks at who is most susceptible to misinformation. A recent study uses a validated online test, the MIST – misinformation susceptibility test, which includes information on education, gender, age and political leanings, to see which demographics are most susceptible. You can take the test yourself here.
In the study they looked at data from 66,242 individuals from 24 countries. They found:
“Multilevel modelling showed that Generation Z, non-male, less educated, and more conservative individuals were more vulnerable to misinformation. Furthermore, while individuals’ confidence in detecting misinformation was generally associated with better actual discernment, the degree to which perceived ability matched actual ability varied across subgroups. That is, whereas women were especially accurate in assessing their ability, extreme conservatives’ perceived ability showed little relation to their actual misinformation discernment. Meanwhile, across all generations, Gen Z perceived their misinformation discernment ability most accurately, despite performing worst on the test. “
These results create more questions than answers, because the why is what is most important. There are a few points of interest. One, there was no apparent “Dunning-Kruger” effect, meaning that overall people who rated their ability to discern fake from true headlines were actually better at it. In fact, Gen Z performed the worst but were the most accurate in their discernment. Women also were below average in performance but above average in discernment.
The most difficult category to interpret, in my opinion, is the political leanings. While the test seemed politically balanced, it’s hard to account for all possible confounding factors. They used, for example, a large language model to generate the headlines, and the training data may have been biased. What I think this means is that we cannot know if there is something inherent about conservatives that makes them more believing of fake headlines, or if current conservative ideology simply aligns more with the kinds of fake headlines that would be created by an LLM trained on the internet. To be clear, I think this is a real effect (because it seems to keep cropping up in research) but it likely reflects the political culture rather than the people.
The fact that the younger generation is more susceptible could simply reflect less life experience, but it may also be a generational effect. Spending lots of time of social media, for example, there appears to be a “post-truth” culture in the younger generation. We may simply be seeing the effects of a generation growing up with social media.
A recent meta-analysis of 31 experiments using the fake headline paradigm found similar results:
“We found that older adults, Democrats (compared to Republicans), and those with higher analytical thinking skills show greater discrimination ability. Ideological congruency (alignment of participants’ ideology with news), motivated reflection (higher analytical thinking skills being associated with a greater congruency effect), and self-reported familiarity with news were linked with a true-news bias.”
This data also finds that being older and more liberal predicts better discriminatory ability. Higher analytical skill also did, which aligns with higher education found in the current study. But also they found that if someone was familiar with the claim being made in the headline they were more likely to believe it (whether it was true or not). This means, unfortunately, that simply repeating misinformation can be extremely effective.
These types of studies are useful, but they are just a starting point. There is also a body of research about countering misinformation. The quick version is – it requires not only fact checking but scientific literacy, critical thinking skills, and media savvy. You essentially have to give someone a new way to understand the world. You can’t just tell them their belief is wrong.
But the bigger question of how we “win” the misinformation wars (whatever a win looks like) is a much tougher nut to crack. Social media companies have essentially given up (and I don’t think their hearts were ever truly in the effort). There is no political will or consensus for any serious measures. And artificial intelligence is about to make the situation orders of magnitude worse.
For now the best we can to is to fight misinformation with good information, and to do what we can to promote scientific literacy and critical thinking skills.


Dr. John Ioannidis Says Obsessive, Self-Anointed “Science-Based” Blamers Are a Total Embarrassment for Science.
The anatomy of a DARVO.
Jonathan Howard on June 30, 2026
Link

While his COVID output and enabling of MAHA doctors should be remembered, it’s not clear why anyone should pay attention to anything Dr. John Ioannidis says moving forward. However, he recently penned an article that provides a good opportunity to discuss a manipulative rhetorical technique known as DARVO. Though it was originally conceived to describe abusive relationships, DARVO has become a common tactic in supposed scientific debate and is worth discussing for that reason.
According to one description:
DARVO is an acronym for manipulative communication: Deny, Attack, Reverse Victim and Offender. Coined by psychologist Jennifer Freyd in the 1990s, DARVO describes how individuals may evade accountability when confronted with their wrongdoings.
DARVO combines elements of gaslighting and blame-shifting. Instead of addressing the issue, perpetrators deny the facts, attack the accuser, and position themselves as the true victim while portraying the original victim as the aggressor.
DARVO is more than just “someone overreacting”—it’s a calculated narrative shift often used by people who benefit from not being held accountable. It serves to evade responsibility, maintain power and control, discredit the person raising the alarm, and centralize sympathy and attention back on the perpetrator.
Dr. Ioannidis’s article was titled Misinformation Blaming and Labelling May Damage Health, Science and Policy Making, which prompted this response from Kevin Kretsch:
Misinformation labelling may damage health, science, and policy making? Gee, that sounds an awful lot like somebody wants misinformation to be left alone and never called out. I wonder why that might be?
That was my reaction too, and Dr. Ioannidis’s article did not disappoint. It contained the following paragraph:
Blaming credentialed, legitimate scientists can also backfire. Rigorous organized scepticism and evidence-based questioning are quintessential for science and desirable. Emotional pre-emptive rejection of scientific arguments may suggest to the public that science is an authoritarian, even retaliatory, process. Some scientists self-anoint themselves as “science-based” communicators and obsessively blame other legitimate scientists with whom they disagree, even without clear evidentiary basis for such accusations. They may attack opponents as quacks. For the watching public, legitimate science becomes indistinguishable from overt pseudo-science. Quackery can only benefit from having serious scientists and serious arguments relegated to the same status. Moreover, when sometimes “science-based” blamers are subsequently proven wrong, in retrospect their obsessive behavior becomes a total embarrassment for science. Science is better served by elevating standards of reasoning and methodological rigor than by engaging in rhetorical escalation or questionable analogies.
At first blush, that sounds pretty reasonable. Let’s examine it one piece at a time.
Blaming credentialed, legitimate scientists can also backfire. Rigorous organized scepticism and evidence-based questioning are quintessential for science and desirable.
I agree with all that. It’s atrocious that MAHA doctors berated and blamed legitimate scientists for literally everything bad about the pandemic, even though Dr. Anthony Fauci’s children needed bodyguards and Dr. Peter Hotez was stalked at home,
These are not the credentialed, legitimate scientists whom Dr. Ioannidis is defending however, and Dr. Ioannidis never directed such scolding at MAHA/MAGA doctors, no matter what they did. They were except from such scolding.
Emotional pre-emptive rejection of scientific arguments may suggest to the public that science is an authoritarian, even retaliatory, process. Some scientists self-anoint themselves as “science-based” communicators and obsessively blame other legitimate scientists with whom they disagree, even without clear evidentiary basis for such accusations. They may attack opponents as quacks.
Is Dr. Ioannidis blaming the credentialed, legitimate scientists here at SBM? It sure seems that way to me, and his accusations of an “emotional pre-emptive rejection of scientific arguments” perfectly describes his paragraph.
Notice, for example, his use of scare quotes to describe “science-based” communicators. He did that to preemptively discredit us and trigger emotions in his readers. The same is true for his allegation that we “obsessively blame… without clear evidentiary basis for such accusations.” That makes us sound like real jerks!
But is it true? Do we make baseless, unfounded accusations?
Dr. Ioannidis provided no examples to support his claim. He didn’t quote us or reference any of our essays. His lamentations were entirely content free. Rather than actually debating the science, Dr. Ioannidis only ever wants to talk about decorum and the format of some hypothetical scientific debate, one he refuses to actually engage in. He previously made these exact empty complaints in an article with Dr. Vinay Prasad. I guess that makes him “obsessive” too.
As Christopher Hitchens famously said, “What can be asserted without evidence can also be dismissed without evidence”. While that alone would be enough to dismiss Dr. Ioannidis’s accusations, fortunately we have an easily accessible track record here at SBM and in my books. We eagerly invite people to read our work and judge for themselves whether our criticisms are “without clear evidentiary basis.”
I think we provide plenty of evidence.
For example, I emailed Dr. Ioannidis before I published my first essay on him here to make sure I was representing his views fairly. He did not respond. Nonetheless, that essay contained lengthy quotes of his combined with detailed explanations of why I disagreed. That’s all I’ve done ever since, and it’s what I’m doing here too. I never just called him names.
Though he obviously has no obligation to do so, Dr. Ioannidis hasn’t responded to any of our good faith criticisms in a good faith manner. While he parades his hurt feelings, wags his finger, and sanctimoniously scolds us for our “obsession,” not once has he raised any meritorious objections to our work, and SBM is not unique in this regard.
Despite his lofty words, praising “rigorous organized scepticism and evidence-based questioning”, Dr. Ioannidis rarely, if ever, engages with his critics in a serious, sober way.  Instead, he acts as if he’s our “opponent” in some game and absurdly implies that we’ve written dozens of lengthy articles that do nothing but call him a “quack.” To Dr. Ioannidis, accurately quoting him and disagreeing in any way constitutes an “attack”. Imagine his outrage if we called him a “fucking moron”, “fucking liar” or “piece of shit,” like Dr. Vinay Prasad. Dr. Ioannidis apparently has no problem with this however. He referenced their prior work on the value of civil discourse in his latest article.
If Dr. Ioannidis quotes our work and explains exactly where we he thinks we erred, it will be the first time. Of course, I’ll keep an open mind to any substantive criticism. Until then, we can recognize he is just obsessively blaming legitimate scientists with whom he disagrees, even without clear evidentiary basis for such accusations.
Simply put, he is attacking us while claiming to be attacked.
For the watching public, legitimate science becomes indistinguishable from overt pseudo-science.
Not only do I agree with this, the disgraceful blending of legitimate science with overt pseudo-science has been a core theme of my writing and of SBM long before I arrived. Dr. Gorski wrote an article in 2018 titled Quackademic Medicine and the Delusion of Being “Science-Based”, and the subtitle to We Want Them Infected warned about doctors who embraced the anti-vaccine movement.
As Dr. Gorski’s series on Lysenkoism 2.0 shows, we here at SBM talk about this every week. In contrast, Dr. Ioannidis has nothing to say about Kennedy and MAHA doctors. While he issues frequent vague warnings about potential, future threats to science, he is silent about and unbothered by what is actually happening to science today. I’ll confess I even asked ChapGPT about this. Here’s what it said:
I could not find a public statement where Ioannidis specifically rebuked Kennedy or called out his positions by name. If such a statement exists, it does not appear to be widely documented.
So, assuming Dr. Ioannidis agrees with us that Kennedy is a quack, here are some examples of prestigious professors from prestigious universities confusing the public by making legitimate science indistinguishable from overt pseudo-science.
Dr. Martin Kulldorff, Harvard University: The Cure for Vaccine Skepticism
Dr. Vinay Prasad, UCSF: Sabotaging RFK Jr’s Confirmation Will Increase Vaccine Hesitancy
Dr. Jay Bhattacharya, Stanford University: RFK Jr Will Disrupt the US Medical Establishment
Dr. Marty Makary, Johns Hopkins University: Dr. Makary On RFK Jr’s Criticism Of The ‘Medical Industrial Complex’: Saying Things ‘People Know Are True’
While Dr. Ioannidis has only criticisms of SBM, he has only praise for such MAHA/MAGA doctors. For example, not long ago he said:
I do worry that taking the stance that ‘Everything was perfect and now comes Jay Bhattacharya and destroys everything’ is very problematic. We need some shaking up and we need that to happen rigorously. If we don’t, it will be done with political means.
Dr. Ioannidis has nothing to say about their actual performance in power, or the state of science under MAGA/MAHA. He might be the only scientist in America with no opinion on the matter.
Instead of honestly grappling with the news headlines he helped enable, Dr. Ioannidis reserves his contempt and scorn for those of us who tried to warn about them. Is his view, our tiny little blog is the big problem. We need to stop being “obsessive.” We need to keep our mouths shut and lay off.
Dr. Ioannidis and MAHA/MAGA doctors
Quackery can only benefit from having serious scientists and serious arguments relegated to the same status.
Of course, people cannot always be easily dichotomized into “quacks” and “serious scientists.” Plenty of serious scientists have made unserious arguments, and despite Dr. Ioannidis’s protestations, they are not entitled a lifetime free pass where they are forever immune from criticism. A Nobel Prize winner who embraces quackery, and there have been several, deserves to be called out for embracing quackery.
For me at least, any doctor who spread COVID misinformation and/or promoted, amplified, and legitimized Kennedy/MAHA in any way has permanently forfeited the right to be taken seriously as a serious scientist ever again, no matter their past scientific acumen or current homages to “rigor” and “evidence.”
Moreover, when sometimes “science-based” blamers are subsequently proven wrong, in retrospect their obsessive behavior becomes a total embarrassment for science.
Once again, Dr. Ioannidis provided no examples of us “science-based’ blamers” being proven wrong, and so once again we can dismiss his juvenile taunts out of hand.
While Dr. Ioannidis makes unfounded allegations about us being “proven wrong,” he vastly overstates our power and influence by calling us a “a total embarrassment for science“, as if with everything that’s happening in the world right now, the entire scientific community is reading SBM articles and shaking their head in collective shame over our “obsessive behavior.”
Meanwhile, back in the real world, STAT News recently published an editorial titled I’ve Spent 40 Years In Research. I Have Never Seen A Threat To Science Like The New Grantmaking Rule. It didn’t mention SBM.
Of course, we are not perfect. I’ve readily acknowledged a couple goofs here and there, though by far the biggest mistake we made was to underestimate the threat to science.  I previously discussed how I bent over backwards too long to give the benefit of the doubt to doctors who spread disinformation to trick the public about COVID, advance their own careers, and promote the MAHA/MAGA agenda.
These doctors are a total embarrassment for science and it is all they and their enablers should be known for moving forward.
Science is better served by elevating standards of reasoning and methodological rigor than by engaging in rhetorical escalation or questionable analogies.
I agree with that too. Dr. Ioannidis can prove he means it by coming on my podcast for some civil debate and discussion. There will be no rhetorical escalation or questionable analogies. We’ll play lengthy clips of his 2020 pandemic pronouncements and provide examples of him boosting MAHA/MAGA doctors. We will listen patiently, without interruption, while he defends his words, and he is welcome to do ask me to do the same. I’m ready to defend it all.
Something tells me his commitment to “elevating standards of reasoning and methodological rigor” doesn’t extend that far, however. It’s a lot easier to insult us as obsessive “science-based” blamers who are a total embarrassment for science and then accuse us of attacking him.
DARVOs are attempt to silence people, and we aren’t going to let that happen here at SBM.


Bad Pharma: A Manifesto to Fix the Pharmaceutical Industry
Scott Gavura on November 22, 2012
Link
“There is no medicine without medicines” write Ben Goldacre in his new book Bad Pharma. To Goldacre, an author, journalist and physician, this cause is personal. The title, a reference to both his first book, Bad Science, as well as the pharmaceutical industry’s nickname Big Pharma, is a bit of a misnomer. While the focus is pharmaceutical companies and their actions, there are a number of enablers in the health care system – medical journals, regulators, and even medical professionals, all of whom have put the industry’s needs ahead of good medicine. According to Goldacre, the damage is pervasive and deep, right to the roots of modern medicine. These problems know no borders, and affect us all. Despite the different health care systems that exist worldwide, we all depend on for-profit pharmaceutical companies to develop and market new medicines. These companies collectively wield enormous clout, due in part to the remarkable success of medicines over the past several decades. The global pharmaceutical market will probably top $1 trillion (yes, 12 zeros) this year. And Goldacre argues the industry is not only compromised, it is broken. And over 400 pages, he defends the following paragraph:
Drugs are tested by the people who manufacture them, in poorly designed trials, on hopelessly small numbers of weird, unrepresentative patients, and analysed using techniques that are flawed by design, in such a way that they exaggerate the benefits of treatments. Unsurprisingly, these trials tend to produce results that favour the manufacturer. When trials throw up results that companies don’t like, they are perfectly entitled to hide them from doctors and patients, so we only ever see a distorted picture of any drug’s true effects. Regulators see most of the trial data, but only from early on in a drug’s life, and even then they don’t give this data to doctors or patients, or even to other parts of government. This distorted evidence is then communicated and applied in a distorted fashion. In their forty years of practice after leaving medical school, doctors hear about what works ad hoc, from sales reps, colleagues and journals. But those colleagues can be in the pay of drug companies – often undisclosed – and the journals are, too. And so are the patient groups. And finally, academic papers, which everyone thinks of as objective, are often covertly planned and written by people who work directly for the companies, without disclosure. Sometimes whole academic journals are owned outright by one drug company. Aside from all this, for several of the most important and enduring problems in medicine, we have no idea what the best treatment is, because it’s not in anyone’s financial interest to conduct any trials at all. These are ongoing problems, and although people have claimed to fix many of them, for the most party they have failed; so all of these programs persist, but worse than ever, because now people can pretend that everything is fine after all.
We all have our own biases, and I should disclose mine. I’m a pharmacist who has seen HIV go from a death sentence to a chronic disease, thanks to newly developed drugs. I’ve watched cancers like leukemias be effectively cured, thanks to medication. And I’m amazed that surgeries like double-lung transplants, impossible in the past, are now a reality, thanks in part to drug treatments. Yet I’ve also spent more than a decade reviewing the efficacy and safety of prescriptions drugs. Regrettably few are truly innovative. Many are approved with lingering questions about long-term safety and effectiveness. The value some offer can be questionable. I’ve also seen tremendous harms caused by drugs – from individual patients who have suffered horrible adverse drug reactions to population-level disasters like the Vioxx (COX-2) debacle. And I haven’t ignored the countless fines levied on pharmaceutical companies for bad, and sometimes even criminal, behavior. With its repeated capacity for self-sabotage, the pharmaceutical industry is its own worst enemy. My colleagues who work in the pharmaceutical industry agree. They’re smart, honest people that genuinely want to help get good treatments to patients. They’re embarrassed by what they see. So while I have no doubts about the astonishing track record of innovative new drugs that have transformed medicine, I also have no illusions that drug companies always behave in ways that support science-based medicine. And I think there is the potential for the industry to do much better. So how do we get this?
Goldacre, a medical doctor by training who is still in active practice, has been writing a column on science and evidence for The Guardian for almost a decade. He hasn’t pulled any punches, calling out bad reporting, taking on media personalities who spout pseudoscience, and even even getting sued for libel by a vitamin purveyor. (Goldacre won.) He compiled many of these themes into his first book Bad Science, one of the best resources for non-scientists interested in learning the practical application of the scientific method. From homeopathy to media nutritionists, Goldacre effectively demystifies scientific inquiry and exposes the tricks that can be used to spin and obfuscate the facts. The pharmaceutical industry makes an appearance in the book as well, which Goldacre uses to point out how it can also manipulate good science, using the same tactics (though perhaps with more sophistication) than alternative medicine purveyors. In his discussion of the industry, he foreshadows the thesis of Bad Pharma:
Whatever our political leanings, everyone is basically a socialist when it comes to healthcare: we all feel nervous about profit taking any role in the caring professions, but that feeling has nowhere to go. Big pharma is evil: I would agree with that premise. But because people don’t understand exactly how big pharma is evil, their anger and indignation get diverted away from valid criticisms—its role in distorting data, for example, or withholding life-saving AIDS drugs from the developing world—and channelled into infantile fantasies. ‘Big pharma is evil,’ goes the line of reasoning, ‘therefore homeopathy works and the MMR vaccine causes autism.’ This is probably not helpful.
Goldacre’s comment captures the trepidation I had before reading the book. Constructive criticism of the pharmaceutical industry is surprisingly difficult to find. Many critics have a clear bias. Big Pharma is evil…and…that’s where it ends. There’s no acknowledgment of the value medicine offers, or the recognition that industry can do better. Big pharma’s failings are a blunt instrument that drive home an unsophisticated, and demonstrably incorrect, point: Medicine itself is bad. Some critics seem to have given up entirely on improving the system: It’s hopelessly rigged and even corrupt, they argue. Lest you think this is solely the domain of CAM providers or advocates like Joe Mercola or Mike Adams, it’s not hard to find academics, and sometimes even medical doctors, whose pharmaceutical industry criticisms are not driven by a desire for better science, but rather reflect a rejection of it. In short, they become cranks.
Reassuringly, Goldacre doesn’t get into black helicopter, medical-industrial complex, conspiracy theories with crank objections to industry. He’s a practicing physician who acknowledges the benefits of drugs but laments a system which has even compromised his own treatment decisions. One of his early anecdotes describes how the publication bias of trials for reboxitine, an antidepressant, gave a skewed perspective of the evidence:
I did everything a doctor is supposed to do. I read all the papers, I critically appraised, them, I understood them, I discussed them with the patient, and we made a decision together, based on the evidence. In the published data, reboxitine was a safe and effective drug. In reality, it was no better than a sugar pill, and worse, it does more harm than good. As a doctor I did something which, on the balance of all the evidence, harmed my patient, simply because unflattering data was left unpublished.
Goldacre’s passion and palpable outrage at the fact this has occurred, and continues to occur, are written with a call to action: The situation can can be fixed. And there’s a larger cause – we can all benefit:
However rich you are, if you are sick you can’t innovate new medicines overnight, because that takes time, and more money than even you have. And you can’t know the true effects of the medicines we have today because nobody does, if they’ve not been properly tested, and if some results go missing in action.
Certainly, compared to Bad Science, this is a much longer, much more detailed read. Goldacre recognizes the need for meticulous accuracy but also strove to make the book readable and understandable to the lay reader. It is a tough balance, but I think it largely succeeds. There are only six chapters, of which the first chapter, on missing data, and the final chapter, on marketing, take up almost two thirds of the book. Each chapter provides some illustrative anecdotes, summarizes the overall evidence base, and then proposes a number of possible solutions – some for individuals, and some calling for system-level change. An afterword sums up his detailed prescription. I’ll take the chapters in the same order.
Missing DataGoldacre starts with missing data, the basis of his thesis and one of the most fundamental problems in medicine today. As I noted in a recent post on on clinical trials transparency, missing data compromises our evaluation of the evidence base, prevents science-based care, wastes resources, and ultimately, does a disservice to patients. If trials are conducted that don’t support efficacy claims, yet are never published, the evidence base becomes biased in favor of treatments and interventions. Goldacre summed up the issue concisely in a recent interview in The Economist:
If I were to run a study, and then just remove half of my data points so that my results looked much better, well, you would laugh in my face. It would be obvious to anyone that it was research misconduct. You might even call it fraud. And yet we tolerate the results of entire clinical trials—a huge proportion of them—being withheld from doctors and patients. In medicine, we rely on summaries of evidence, we collate the results from many different trials. So withholding the results of whole trials is exactly the same insult to the data as fraudulently deleting data points from within individual studies.
There is ample evidence that the evidence base we are currently using in medicine is distorted exactly as Goldacre describes. However, the extent to which this has harmed our treatment evaluations is probably impossible to discern. Goldacre points out that this isn’t just pharma’s fault – research ethics boards, universities, regulators, and medical journals have all failed to follow-through on new standards for data transparency and access. Goldacre calls these “fake fixes”, such as the requirement from the world’s top medical journals to ensure that all published trials are registered before they commence – but the requirement isn’t subsequently enforced. These fake fixes extend to regulators who offer lip service on transparency, and even block access to the evidence they have used to determine if a drug should be approved for sale. Instead there is a “yes” or “no” decision, with little public sharing of the evaluation process and information considered. Goldacre makes an important point about regulatory decisions versus those made by health care professionals:
Contrary to what some regulators seem to think, a drugs is not either ‘good’ and therefore on the market or ‘bad’ and off it. A regulator makes a decision about whether it’s in the interests of the population as a whole that the drug should be available for use, at all ever – even if only is some very obscure circumstance, infrequently and cautiously. The bar is set pretty low, as we shall see, and lots of drugs that are on the market (in fact, the overwhelming majority) are hardly ever used.
A doctor needs to use the same information as that available to the regulator in order to make a very different decision: is this the right drug for the patient in front of me right now? The simple fact that a drug is approved for prescription doesn’t mean it’s particularly good, or the best. In fact, there are complex decisions to be made in each clinical situation about which drug is best.
Goldacre concludes the chapter with an extended list of fixes, outlining ways to strengthen the checks and balances to ensure that the results of all human trials are made publicly available. Some are major policy issues, others are much more simple to implement. Would they improve transparency? Undoubtedly. Are these new issues? No. But the culture of complacency that exists in the health care system about this issue has so far been remarkably resistant to meaningful change.
Where do new drugs come from?The second chapter looks at the drug development process, and covers a huge subject in a handful of pages. Consequently the complexity and risk of drug development is simplified, which is unfortunate, as it’s important to keep in mind that we have collectively decided that much of this risk must be carried by these for-profit companies. A later mention of drug development costs is also abbreviated, which again minimizes the financial consequences of development. (See my old post on this issue for a detailed discussion of drug development costs.)
One of the most interesting sections is a discussion on just who is participating in clinical trials. Until the 1980’s, many trials in the United States were being performed on prisoners. Today, it’s increasing likely that the crucial trials supporting the approval of a drug are being conducted overseas, where it’s apparently much easier, and less expensive, to run a trial. The challenges with ensuring effective ethical standards and clinical trials oversight should be obvious. Some of the problems with poor countries accessing these drugs are discussed, and Goldacre explores the ethical issues with studying drugs in patients who will never be able to afford the treatments.
Bad regulatorsThe overall purpose of clinical trials, from a manufacturer’s perspective, is to collect information to support a drug’s regulatory approval. If you’re a frequent reader of this blog you’ll know that there is no shortage of criticism of regulators for the work they do in evaluating new drugs. One of the biggest is the challenge of obtaining expert opinion on new drugs from individuals who may have significant conflicts of interest. This is a common problem, where those with the most experience using a drug, are often in direct conflicts of interest, most frequently because of compensation for participation in clinical trials. The conflicts that can exist between regulators and the industry it regulates are real, and probably managed less transparently than we would like.
Another criticism Goldacre raises is the problem with accepting surrogate markers as endpoints in trials. Manufacturers want their drugs approved as quickly as possible (the patent clock is ticking) so often ask to measure what are called “surrogate” indicators, which are proxies for “hard” indicators which are the outcomes we are really interested in. For example, blood pressure, or cholesterol measures may be the endpoint measured in a clinical trial, rather than a more meaningful indicator like heart attack, stroke, or death. Measuring surrogates is easy. Measuring hard outcomes is time consuming and could mean the need to study hundreds more patients. When surrogates are demonstrably accurate predictors of hard outcomes, we’re in a good place – it’s fair to use a short, smaller trial. But assuming a surrogate predicts a hard outcome can lead to bad decisions. The classic example is the CAST trial, which tested a group of heart drugs for their ability to control abnormal heart rhythms – the surrogate. But the trial also measured hard outcomes. And the trial answered the question – the surrogate improved, but the treatments were killing, not helping patients.Goldacre gets into the problem with so-called “me-to’s” which are variations of a competitor’s drug product. For example, there are several different but similar “statin” drugs on the market, as there are a group of similar drugs for blood pressure called “ACE inhibitors”. The typical pharma criticism is to berate industry for lacking innovation and playing it safe. But this is a view shaped by an ignorance of the pharmaceutical development process. First, these drugs are often all in development at the same time. Once a potential pathway or drug target has been identified, it’s a race to design drugs, complete the trials and be first on the market. And until the trials are done, and the drugs are approved, it’s anyone’s guess which is going to be the “new” drug and which will be the “me-too”. So we may have a dozen or more similar drugs on the market. That’s a consequence of a competitive marketplace. I share Goldacre’s mixed feelings about me-too’s. This kind of variety is useful to patients who may prefer one over another. Yet it’s also true that the pharmaceutical industry really doesn’t compete on price, nor does it regularly conduct head-to-head trials comparing drug A versus drug B. So we end up with markets where there’s little clear information on which drug is best, and worst of all, no price competition, which would at least reduce the costs of treatment. Goldacre points out the pressing need for comparative effectiveness research, studies which compare treatments and help us determine which is superior.One of the most important roles of a regulator is the requirement to monitor its safety after approving it for sale. Given a drug is usually tested in small numbers of patients who are carefully selected and closely monitored, there can be a number of surprises once a drug is unleashed on a market. I covered a lot of this ground in a prior post on adverse event reporting. Goldacre spends time describing how regulators worldwide do a generally poor job of both collecting safety information, but also sharing it with health professionals and the public. Some of the recommendations he has to improve regulation includes:
    • requiring companies to compare new drugs against the current “standard of care” rather than placebo
    • raising the evidence bar to demand trials that answer more relevant questions useful for patients and physicians
    • dramatically increasing the transparency of drug evaluation process, so that the evidence supporting decisions can be evaluated
    • improving the way we share information on the risks and benefits of drugs
All of these solutions are achievable. Transparency and data access are obvious. There is little cost and tremendous upside. Changing the standards for clinical trials would have a much more profound effect on drug development, possibly delaying new drugs for years. Goldacre argues this is justified – we’re not losing out if there’s no evidence to demonstrate that the new treatment is better than the current standard. It’s an important policy issue which merits discussion.
TrialsTwo chapters are devoted to the structure of clinical trials – how they’re often flawed, and how biases can work their way into the evidence. Patients may be carefully selected for a study and they may be largely unrepresentative of the real population that ultimately take the drug. So new medication that appears to offer a substantial benefit may offer little improvement when it’s used in the real world. Unfortunately, this happens all the time. Or trials may not even answer relevant questions we might have: Studies may be too short (surrogate markers again) or they compare a drug to something inappropriate (placebo, perhaps, instead of the current best treatment). The chapters are essentially a quick-and-dirty critical appraisal guide, and Goldacre gives example after example of the spin that can occur. One of the interesting projects that Goldacre describes is his work in promoting real-world evaluations of effectiveness, which have the potential to tell us which drug works best in a representative population. Clearly he’s excited by the possibility of these pragmatic studies to answer questions that have (so far) been unattainable.
MarketingThe last chapter is devoted to marketing in all its guises. Goldacre describes it as a “chaotic and biased” system, which is a fair description. Regrettably most countries rely on pharmaceutical companies to disseminate information on their own products – and they are hardly unbiased sources of information. One of the challenges we have in the way drugs are prescribed and distributed is that we have a set of misaligned incentives that create market failure. These incentives are easily manipulated through marketing. Consider the following:
    • The patient wants the best treatment, and generally trusts the physican’s decision-making. Their opinions may have been influenced by direct-to-consumer advertising. In some cases, they may believe they are ill because of advertising. Costs of treatment may or may not be a concern, depending on insurance coverage.
    • The physician prescribes the drug. The decision making process for physicians is not always rational, or evidence-based. They may have been exposed to biased sources of information about a drug’s merits. The cost of the drug may not be known – and the physician may not care.
    • The pharmacist has verify the safety and appropriateness of the drug, but well after the treatment decision has already been made. The question of “who pays” has to be sorted out. There is usually little opportunity to help guide drug selection. The pharmacist may also be a target of direct marketing from pharmaceutical companies.
    • The insurer (private, government, or otherwise) will often know little about the individual patient circumstances. They generally want the same thing as the patient – the most appropriate therapy – but know the cost of each treatment and will have a preference for using more cost-effective treatments first.
Take for example, the now infamous case of the COX-2 drugs (Vioxx and Celebrex) when they were launched. Here were drugs that seemed to offer marginal but promising benefits (fewer gastrointestinal bleeds) in selective patients based on short trials. However, they were significantly more expensive that other anti-inflammatory drugs. Yet massive marketing created both patient demand and physician prescribing, accelerated by manufacturer-sponsored “continuing education” programs, and unbelievably, even fake journals created to give the impression of academic credibility. And insurers, who might have preferred less expensive anti-inflammatory drugs, had to make decisions about whether to restrict funding, or pay a tremendous premium for the potential minor benefits of these new treatments. These drugs went on to be blockbusters – but with tragic consequences.
It’s the overall issue of misaligned incentives that I would have liked to see explored further in Goldacre’s book. The United States is 45% of the global pharmaceutical market, followed by Japan (9%), Germany (5%) and the United Kingdom, at 4%. For-profit companies will act to maximize their revenue in the biggest market. It’s arguable that the United States has the least cost-efficient health care system in the world. As long as that system continues to allocate health resources in inefficient ways (e.g., paying for drugs that don’t offer meaningful new benefits), can we really expect pharmaceutical companies to change on their own? Regulation is important, but incentives drive behavior, and we should reward companies who bring the truly innovative therapies to market.
Again, Goldacre writes a number of prescriptions for improvement. They range from the simple (“Don’t see drug reps”) to bigger policy issues about how we can improve the continued professional development of physicians. All merit consideration. And there’s no question, that if implemented, they would dramatically improve the way we use medications.
Closing thoughtsThis is a book with important messages for patients, health care professionals, governments, and the pharmaceutical industry. It may anger, and even infuriate you. It should. Yet Goldacre sketches out a path for reform – one in which transparency is the rule, not the exception, and one in which the pharmaceutical industry can engage with the health care system in ways that will help it do what we all want it to do – bring innovative new medications to market that improve the health and welfare of patients. This change, however, will only come when we demand better – better from pharmaceutical companies, better from regulators, and better from our fellow health professionals. Ultimately, we’ll have the industry we deserve – if we’re not willing to demand better, we’re never going to see any improvements. This book is your call to action.

The Book “The War on Science” Reveals How 39 Renowned Scientists and Scholars Enabled the War on Science
While many “progressives” tried to warn about Trump’s threat to science, these renown scientists and scholars sat on the sidelines and undermined them.
Jonathan Howard on August 1, 2025
Link
Trump’s Anti-Science Campaign.
In 2016, the physicist Lawrence M. Krauss, wrote an article titled Trump’s Anti-Science Campaign. It was astonishingly prescient. As if he had a crystal ball, Krause wrote:
Trump has argued for downsizing the Department of Education and said that the U.S. invests too much money in K-12 schooling. He has suggested that he might appoint Ben Carson—a young-Earth, anti-evolution creationist—to advise him on educational reform. He has called the National Institutes of Health “terrible,” and has said that he would eliminate the E.P.A. In April, the science journal Nature reported that his anti-immigrant tirades could be hindering efforts to recruit good scientists and students to the U.S. The list goes on…
The positions taken by Trump and the Republicans have consequences beyond science itself. Essentially, they are betting that, for a significant portion of the country, empirical reality doesn’t matter; they are also signalling that empirical reasoning won’t be the basis of their public policy. Today, of course, we face global challenges such as climate change, which are more urgent than any we have ever confronted. These challenges require a sober assessment of reality. When science is distorted on the campaign trail, it may produce applause lines. But if those distortions lead to bad public policy, the quality of people’s lives will suffer.
People suffered alright, and we are just getting started.
The bright side is a likely boost free speech and due process at unis
However, shortly after that prophetic essay, Krauss’s actions caught up to him in a big way, and instead of quietly retreating from the public eye, he directed his grievance at a different opponent. By the time Trump 2.0 rolled around in 2024, Krauss saw a “bright side”. He called Trump a “nut, liar, and crook”, but said “the bright side is a likely (to) boost free speech and due process at unis.“
Things didn’t work out that way, and Krauss rediscovered that Trump wasn’t a friend of free speech and due process at unis after all. He recently wrote an article on Quillette titled Trump’s War on Science that said:
I’d be one of the first to admit it: higher education in the US is in urgent need of significant reform. Harvard University, for example, discriminated against Asian applicants, especially Asian men, and allowed vicious antisemitism to run rampant on the grounds that it was a necessary part of free speech—all the while rigorously policing speech of which it disapproved and punishing faculty whose research results suggested that racism might not be responsible for all of society’s ills, or who simply stated openly that there are only two sexes. In addition, while discriminating against some talented students and scholars on the grounds of their race and/or sex, the university promoted people on the basis of identity politics whose academic accomplishments were unimpressive and whose plagiaristic work violated academic standards. Our major universities created bloated bureaucracies ideologically coupled to claims of social justice and allowed those overpaid bureaucrats to stifle free speech, academic freedom, and open inquiry.
STEM fields were not exempt from the madness. The National Science Foundation, the Department of Education (DOE), the National Institutes of Health (NIH), and NASA all became so wrapped up in being “anti-racist” that key aspects of their scientific mission were suffering. Indeed, the situation had become so worrisome that I joined 38 distinguished faculty who span a variety of different fields and political allegiances in writing about this in a book: The War on Science, which will be released this month.
But in response to this internal war on scholarship that has been undermining academic excellence, a new external war has erupted that may prove even more damaging to the economic health and security of the US, and to the future of scientific research and innovation at the country’s universities and scientific institutions.
I stopped reading after that. It’s possible I would agree with everything else Krauss wrote, but I wouldn’t learn anything. Someone who spent the past few years warning about threats to “free speech, academic freedom, and open inquiry” from those concerned about “social justice” has nothing to teach me or anyone else about what’s happening now. It takes no special skill to list the ways Trump is attacking science, and Krauss has no unique insights or wisdom to offer. That he says Trump is a part of a “new external war” shows he forgot his own 2016 essay.
The War on Science: Thirty-Nine Renowned Scientists and Scholars Speak Out About Current Threats to Free Speech, Open Inquiry, and the Scientific Process.
Krauss’s new book is titled The War on Science: Thirty-Nine Renowned Scientists and Scholars Speak Out About Current Threats to Free Speech, Open Inquiry, and the Scientific Process. That sounds promising, and had I written a chapter, it’s no secret what I would have said. Nothing like that appears in the book, however. The War on Science is silent about the war on science.
The War on Science describes itself thusly:
An unparalleled group of prominent scholars from wide-ranging disciplines detail ongoing efforts to impose ideological restrictions on science and scholarship throughout western society.
From assaults on merit-based hiring to the policing of language and replacing well-established, disciplinary scholarship by ideological mantras, current science and scholarship is under threat throughout western institutions. As this group of prominent scholars ranging across many different disciplines and political leanings detail, the very future of free inquiry and scientific progress is at risk. Many who have spoken up against this threat have lost their positions, and a climate of fear has arisen that strikes at the heart of modern education and research. Banding together to finally speak out, this brave and unprecedented group of scholars issues a clarion call for change.
Topics include: Free speech, victimhood, ideology, corruption of academic disciplines, cancel culture, DEI, gender, and race, and what we can do.
“Higher education isn’t what it used to be. Cancel Culture and DEI have caused many to keep their mouths shut. Not so the authors of this book. This collection of essays tells of threats to open inquiry, free speech, and the scientific process itself. A much-needed book.”—Sabine Hossenfelder, Physicist and Author of Existential Physics: A Scientist’s Guide to Life’s Biggest Questions
The table of contents are below.
This must-watch video provides an introduction to the book, its publisher, and the problematic backgrounds of several of its authors, as does this excellent article by Dr. Joshua Weitz.
I don’t know all of these authors, but I am familiar with many of them. Some have always been vile. According to one article titled Penn Law’s Amy Wax Doubles Down on Racist Comments:
“Given the realities of different rates of crime, different average IQs, people have to accept without apology that Blacks are not going to be evenly distributed throughout all occupations. They’re just not, and that’s not a problem. That’s not due to racism,” Wax said in an interview with Concordia University professor Gad Saad on Jan. 24.
Others however used to be widely-admired scientists and science communicators. Yet, aside from a few pro-vaccine Tweets, they were AWOL combatting COVID disinformation and the broader threat it posed to science. None of them were in the trenches warning about Kennedy, his crew, and the wealthy right-wing forces who backed them. I never saw them lament the threats against Fauci and Hotez. That didn’t seem to bother them at all.
Ensconced the their pundit fantasy world, they had the luxury of pretending that the real threat to science came not from Trump and his minions, but rather from “cancel culture, DEI, gender, and race”. While Krauss’s 2016 essay predicted the future, his current book was written as if the past 8 years never happened, and it should be titled Renowned Scientists and Scholars Who Were in a Coma.
The War on Science is best thought of as a work of science fiction, dispatches from a parallel universe where MAGA doesn’t exist, Wokism is all-powerful, and science was obliterated by DEI and trans people. It’s a complete inversion of what’s actually happening.
Cancer Screening Panel Too ‘Woke’ for RFK Jr.
However, it’s not just that these renowned scientists and scholars created a fantasy world, their imaginations provided fuel and ammunition to the people who are currently taking a wrecking ball to things in the real world. Many key Trump officials, past and present, got into power by portraying themselves as woeful victims of censorship and cancel culture (Marty Makary. Vinay Prasad, Jay Bhattacharya, and Kennedy). These Trump officials weaponized their perceived victimhood to distract from their disinformation, attack respected scientists, and bash the institutions they are now trashing.
Predictably, these renowned scientists and scholars were happy to lend their legitimacy to this feigned victimhood. Jordan Peterson and Gad Saad recorded podcasts with future Trump officials about “silencing the opposition” about “academic freedom“. Now that they are in power, these same Trump officials are leading the way, purging, censoring, and defunding scientists. Saad’s chapter is titled Universities as Dispensers of Parasitic Ideas. As Trump crushes universities, Saad wants people to think these ticks, leeches, and mosquitoes deserve their fate.
Indeed, much of the destruction today is purportedly being done to combat the grave dangers these renowned scientists and scholars warned about. Today, they headlines read Cancer Screening Panel Too ‘Woke’ for RFK Jr, A Researcher With Hearing Loss Got A Grant To Study Restoring Hearing. The Trump Administration Canceled It Because Of DEI, and NIH Cancels Sickle Cell Research Grant, Citing DEI. As Christina Pagel wrote in her article Donald Trump’s ‘War on Woke’ is Fast Becoming a War on Science. That’s Incredibly Dangerous:
Donald Trump’s attacks on diversity, equality and inclusion (DEI) initiatives since his January inauguration have been intense, indiscriminate and escalating. A tragic plane crash was baselessly blamed on DEI. All DEI programs within public bodies have been ended and private contractors face cancellation if they also don’t comply. Webpages that defend religious diversity in the context of Holocaust remembrance have been taken down.
Science and academia have been particularly targeted. Universities are threatened with losing federal funding if they support DEI. Government reports and government-funded research are being held back if they include prohibited terms such as “gender”, “pregnant person”, “women”, “elderly”, or “disabled”. Grants funded by the National Institutes of Health are being cancelled if they address diversity, equality or inclusion in any form.
This is what these renowned scientists and scholars enabled.
Watch this video here
Read this article here.
The threats from the Left continue, and that’s for sure
However, this isn’t just about the past. This an ongoing process. That these renowned scientists and scholars are aggressively promoting their book, shows that they have no problem continuing to lend legitimacy to Trump. They remain complicit, and either they don’t know this or they don’t care.
Although Krauss and several other renowned scientists and scholars now grudgingly acknowledge Trump’s war on science, they can’t do so without simultaneously whining about “the Left“, even though such rhetoric is being used as a pretext to attack science right now. Richard Dawkins, for example said that “cutting science budgets is sheer spite“, but he also whinged about “idiotic woke nonsense, ‘gender’ studies.” Jerry Coyne, similarly wrote:
Now I know what you’re gonna say: the 39 chapters in the book below (find it here on Amazon) deal exclusively with threats to science from the Left but, as we all know, at the moment the threats to science from the Right (aka, the Trump Administration) are far more serious. In the short run that may be the case, but in the long run, well, who knows, but the threats from the Left continue, and that’s for sure. So think of it as a bunch of scientists and other academics analyzing how our trade is being hurt by “progressives.”
The threats from the Left continue, and that’s for sure. That’s is pure fantasy. “Progressives” are the least powerful and most vulnerable people in America today. Scientists aren’t fleeing the US because of them, and anyone who truly cared about Trump’s threat wouldn’t waste a single word dunking on them at this moment. To convince us “progressives” were scary, Coyne had to conjure an imagined future where MAGA is neutered, Wokism is all-powerful, and DEI destroys science. In the long run, well, who knows.
Predictably, Coyne doesn’t blame himself for being late to recognize Trump’s threat. Rather he says Trump’s actions were too late for the book. He wrote:
And, at the time we submitted our manuscripts to editor Lawrence Krauss (who added a nice introduction), Trump hadn’t yet started slicing federal grant money from “bad” universities (I see that Penn and Columbia have just caved).
Dawkins similarly lamented that Krauss’s anti-Trump essay was “too late for our book“. To absolve themselves of their blindness to the obvious, these renowned scientists and scholars act as if Trump’s attack on science was some freakish, unpredictable natural disaster. How were they supposed to know?
Of course, reality wasn’t too late for their book. These renowned scientists and scholars were late to recognize reality. My books discuss Trump’s threat. It wasn’t “too late” for myself and many others. Seeing these renowned scientists and scholars belatedly catch up is like watching a baby grasp object permanence. For them, Trump’s risk became real only the moment they could no longer deny it, not one second sooner.
Yet even now, these renowned scientists and scholars casually acknowledge Trump’s threat only to promote their book. This requires them to complain about “progressives“, many of whom were light years ahead these them in recognizing Trump’s threat and trying to combat it. While many “progressives” stuck their necks out, these renowned scientists and scholars sat on the sidelines and undermined them.
Is Free Scientific Inquiry Under Threat?
So no, I wont read The War on Science. Even if contains some valid points, they are completely irrelevant, like being warned about a broken taillight as my car careens over a cliff. There is no reason why anyone should care about the flaws of DEI trainings, real or imagined, in 2025. None of it matters.
And though I don’t think anyone should read this book, its mere existence has great value. It both explains and memorializes how we got to this sad moment. Many renowned scientists and scholars, some of whom should have been valuable allies, were blind to the real danger until it was too late. While many of us warned about the threat to science, they rolled out the red carpet to its attackers, giving them intellectual cover and legitimacy. They did this in the name of reason, free speech, and skepticism, all to combat pronouns and gender studies, which were supposedly existential threats to the entire scientific enterprise.
I also won’t listen to Krauss’s upcoming talk Is Free Scientific Inquiry Under Threat?  We here at SBM have been providing the answer for years, and we have nothing to learn from those who are just now discovering it in real time. I would suggest that instead of giving his talk, professor Krauss and all of these 39 renowned scientists and scholars, take a timeout and exercise their right to remain silent. This may surprise them, but they don’t have to comment on everything, and no one has anything to learn from them right now. If any of them genuinely want to oppose Trump’s war on science moving forward, they need to do so full throttle, not just in passing as they once again moan about “idiotic woke nonsense.“
They also need to learn about all they’ve missed the past few years. Krauss’s article from 2016 is an excellent starting point, but they have a lot of of catching up to do. Fortunately, SBM and related sites have done the hard work already. The warnings were there for them on a silver platter all along. Yet instead of paying attention, they prioritized their contrarianism and enabled everything that’s happening now. That’s the legacy of these renowned scientists and scholars.
At least I won’t have to write a DEI statement ever again.


   Ethics, Law, Education and Media

   Science, Reason, Ethics, and Modern Medicine Part 1: Tu Quoque and History
Kimball Atwood
Link
Several weeks ago I argued here that a physician’s primary ethical obligation is to science and truth. In retrospect I probably should have put it a slightly different way: a physician’s primary ethical obligation is the same as everyone else’s. It is to honesty and integrity. For physicians, however, that means being true to real medical knowledge, among other things, and real medical knowledge comes from science. That is what this and the next several posts will be about.*
First, a little Tu Quoque
After reading some of the comments that followed my posting of two weeks ago, I reluctantly thought to add a few words about the medical profession’s view of physicians selling drugs. It felt tiresome to have to address the issue, because it is beside the point. The series was about “naturopathic medicine,” not about modern medicine. If readers who understand the point will excuse the interruption, I’ll quickly attempt to explain why by posing two extreme possibilities: If MDs are entirely innocent of the relevant breach of ethics, what would that have to do with naturopaths selling drugs? But if MDs are entirely guilty, two wrongs don’t make a right—demonstrating the same irrelevancy.
That is why the ”you should talk” sneer is known, in debate, as the tu quoque (“you too”) fallacy. It’s funny how parents seem to recognize it when faced with children who, in seeking permission to engage in dubious activities, invoke the parents’ own sordid histories or the equally irrelevant, alleged prerogatives of other people’s children. Yet the same parents appear to forget it in other contexts.
I was also weary and wary of those who would draw me into a strawman debate pitting the medical profession against any group of sectarian health advocates. I have only a small sense of “solidarity” with the group of people who have MD degrees, and even less so with organized medicine. My first allegiance, as I’ve explained elsewhere, is to science and reason. Those modes of inquiry, together with their obvious bearing on the integrity of all claims about nature, are the bases for my objections to naturopathy and to pseudoscience in general. The medical profession per se is related but not central to the issue. While thinking about these things, it dawned on me that a discussion of why that is might be useful.
Another tu quoque argument popular among advocates of implausible medical claims is that “medicine doesn’t know everything,” i.e., that there are still diseases left unsolved that people suffer and die from, which somehow impugns modern medicine and the basis for its best practices (science). That said, advocates want to invoke “alternative” medicine, which would be an obvious non-sequitur even if such impugning were justified, which it is not. It is the assertion that science-based medicine can’t deliver—demonstrating an extreme ahistoricity on the part of its advocates—that is the topic of this post. I’ll get to the issue of selling drugs another time.
An Essential, Very Short History of Science-Based Medicine
Economics aside, science and reason, including rational evaluations of empirical observations, have been the bases for every purposeful advance in our species’ age-old struggle with diseases, trauma, specific malnutritional states, anatomic and genetic anomalies, problems of aging, and every other problem that comes under the heading “medicine and public health.” Even before the advent of biology or medical science, empirical observations yielded occasional, useful discoveries, such as cinchona bark for marsh fever, innoculations of pus from cowpox blisters to prevent smallpox, limes for sailors to prevent or cure scurvy, food preservation and preparation by salting, cooking, smoking, cooling, or fermentation, and more. There were, undoubtedly, at least a few such advances in every place where humans lived for any length of time. The adjective “purposeful” is intended to exclude happy accidents of biology and the environment: reason did not create cinchona or vaccinia or salt or fermentation or water or fire, but it certainly led us to discover and use them.
Here are a few notions that have had no bearing, whatsoever, on advances in medicine and public health, and never will: vitalism, mysticism, demonology, voodoo, homeopathy and all other forms of sympathetic magic, qi, meridians, the five phases, the four humors, the five pranas, mysterious “energies,” astrology, psi, chiropractic “subluxations,” “craniosacral rhythms,” fanciful homunculi on the iris or the external ear or the sole of the foot, faith healing, other forms of magical thinking, and many more. To the extent that real advances may occasionally have occurred in contexts of pre-scientific practices, they were due to empirical observations, not to the “theories” themselves. More commonly, particularly in recent decades, such practices and beliefs have merely obstructed useful approaches.
Obviously we haven’t solved every medical problem yet. We’ve barely begun, in the greater scheme of things, yet we’re making progress at breakneck speed. Our species, homo sapiens sapiens, is about 200,000 years old. Our brain, the organ that allows us to reason and thus to solve medical problems, has been in its biologically modern state for at least 40,000 of those years (I remember reading somewhere), and possibly for the entire 200,000. Yet we have begun to solve medical problems in earnest only in the past 150.
A smattering of scientific bases for medicine, including systematic attempts to describe and catalog diseases, existed in antiquity. The investigations and habits that eventually culminated in science-based medicine, however, seem to have begun about 1000 years ago. Evidence for this is a rational treatise on testing medical substances, the first post-mortem dissections, by Ibn Zuhr, early investigations into physiology, and probably more. I am not a historian of medicine and I’ve not read the primary sources for the linked pages in Wikipedia; they strike me as exaggerated (how, hundreds of years before the invention of the microscope and the discovery of micro-organisms, could “bodily secretions…contaminated by ‘foul foreign earthly bodies’ ” be construed as “bacteriology and microbiology“?). Nevertheless, the assertions linked above appear to be justified by actual documents.
In the 16th century Vesalius published the first accurate human anatomy. Within a century William Harvey had described the circulation of blood, and shortly after that Antonie van Leeuwenhoek discovered micro-organisms. All the while, investigations of diseases became increasingly rigorous and reproducible because of post-mortem examinations (autopsies), so that “clinico-pathologic correlations“ were possible. The contemporary scientific revolution, in particular the rise of empiricism, led to other discoveries that set the stage for investigations in the 19th century that heralded the arrival of modern, science-based medicine.
These included the discovery of oxidation and the demonstration that living organisms obtain their energy by that mechanism, in accord with the same thermodynamic obligations that apply to combustion and other chemical reactions, microscopic pathology, cell theory, biochemistry, experimental physiology and experimental medicine, including early insights into the existence and functions of hormones, water and electrolytes in tissues, the excretory role of the kidneys, the nervous system, digestion, respiration, including the oxygen transport function of hemoglobin, the heart’s pumping function, pharmacology, statistics, inhalational anesthesia, local and regional anesthesia, the germ theory, immunology, modern surgery, epidemiology and rational public health measures, X-rays, and more. Among other effects, these fields convincingly refuted such beliefs and practices as vitalism, spontaneous generation, miasmas, the humoral theory of disease, homeopathy and other forms of sympathetic magic, bloodletting, purging, scalding, and more.
Thus real science-based medicine has been around for less than two tenths of 1% of the time that our species has existed. In that short time it has solved most of the major plagues that had previously wreaked death and sorrow upon all humans since time immemorial, and that still do so in regions where, for political or economic reasons, science-based medicine has not become widely available. It has also solved many non-epidemic infectious diseases, all of the major nutritional deficiency states, has discovered almost all of the causes and most of the cures of anemia, has provided safe, effective, and painless surgical cures for common, previously deadly diseases such as acute appendicitis, infantile pyloric stenosis, acute cholecystitis, and acute diverticulitis, has solved the most common type of peptic ulcer disease, has drastically reduced the dangers of childbirth and infant and childhood mortality (previously the sources of the greatest sorrow and suffering that our species has endured), has devised marvelous treatments for severe trauma and burns that until very recently would have meant certain death or severe disability, has solved many of the congenital anomalies that also would have meant certain death or disability, has transformed the lives of patients with previously deadly diseases such as type I diabetes and AIDS, allowing them now to thrive for years, has made other common chronic diseases, such as gouty arthritis, rheumatoid arthritis, and inflammatory bowel disease much more bearable, has made significant advances in diseases such as coronary artery disease, strokes, and cancers, that have become more prevalent now that people live longer, and much, much more.
It’s astounding to consider the tiny sliver of time during which these changes have occurred. When I was a kid in the 1950s, there were plenty of people still living who could remember life before effective, rational medicine and public health. My great-grandfather had been born in 1853 into a family of 5 children, of whom 2 died in infancy. His mother died when he was 12. That was the norm for the mid-19th century in the United States. Widespread suffering, disability, and premature death have almost certainly been the norms for every population of humans throughout the entire history of our wretched existence, and are still the norms in parts of the world that remain too poor to have reaped the benefits of science-based medicine and public health.
Here is another way of considering how recently all of this has happened: if we assume that a generation of humans corresponds to about 20 years (probably too short here and now, but not for most of our existence), there have been about 10,000 generations so far. All of the breakthroughs mentioned above, and more, have occurred during the last 6-8. More recently, within the past 2-3 generations, the stage has been set for even more astonishing breakthroughs. Molecular biology and its applications to genetics, virology, immunology, and pharmacology, for example, inevitably lead to breakthroughs in the understanding and treatment of viral diseases, cancer, hemoglobinopathies such as sickle cell disease, and more.
An example of this is the history of our understanding of AIDS, which progressed from a new, terrible, rapidly terminal disease of unknown etiology, when it was first described in the early 1980s, to one explained in complex detail, with treatments so effective that the majority of patients receiving them could expect to feel well indefinitely, barely 15 years later. Come to think of it, within a period of a few years beginning in about 1980, at least 5 serious diseases—AIDS, Legionnaires disease, Toxic Shock Syndrome, Lyme disease, and peptic ulcer disease—were solved or mostly solved. The first three of those were entirely new and scary when they first presented themselves. Imagine what our ancestors would have thought. Imagine how they might have “treated” such diseases, merely a few generations ago.
And yet as recently as 10 years ago, AIDS was a favorite playground for quacks (still is, but now the same claims are mainly for “complementary” effects). Funny: that was also true for polio, cholera, yellow fever, and every other epidemic disease prior to its being solved by science-based medicine.†
References
†Young JH. American Health Quackery. Princeton: Princeton University Press, 1992

   Science, Reason, Ethics, and Modern Medicine, Part 2: the Tortured Logic of David Katz
Kimball Atwood
Link
In Part 1 of this series* I asserted that a physician’s primary ethical responsibility is to honesty and integrity, which in turn must be largely based on science and reason (I apologize if that sounded preachy; if there had been more time I might have couched it in more congenial terms). I mentioned the fallacious reasoning whereby proponents of implausible medical claims (IMC) point to real and imagined weaknesses of modern medicine to justify their own agenda. I offered, as a favorite example of such proponents, science-based medicine’s having not yet solved every health problem. This week I’ll show how this version of the tu quoque fallacy has led a prestigious medical school to advocate pseudoscience-based medicine.
Modern Medicine: a Brief, Fragile Commitment to Science
First, a few more words about the title of this series. Modern medicine is not science, even if it draws upon science for its knowledge: it is an applied science similar, in that sense, to engineering. Modern medicine is also not synonymous with the “medical profession,” if the term means the collection of all people with MD degrees. That is true for the obvious reason that medicine is more than people, but also because a small but loud minority of MDs rejects modern medicine and science.
Modern medicine has made an uneven commitment to science and reason. At its best, it has formally embraced them in the faculties and curricula of medical schools, in its codes of ethics, and in its contributions to knowledge, both basic and applied, over the past 150 years or so. As discussed last week, it is because of science and reason that modern medicine has made dramatic, revolutionary advances in a very short time. That is what distinguishes it from every other “healing tradition,” and why there is no legitimate competition. The only valid medicine in the modern world is science-based medicine—not “allopathic,” “Western,” “conventional,” “regular,” “integrative,” “complementary and alternative,” or any of the so-called “whole medical systems.” The pre-scientific (and, ironically, “post-modern”) designation of “schools” or “systems” of medicine, so stridently trumpeted by quacks, is an anachronism—even if it persists in archaic, governmental edicts.
Compared to the actual sciences, however, modern medicine’s commitment to science is fragile. Its recent confusion of error-prone clinical trials with science itself—the project called “evidence-based medicine”—has been a mixed blessing. Its growing tolerance of charlatans and crackpots, at times elevating them to celebrity status, would be unthinkable in physics or biology. Its dalliances with quackery, so depressingly recounted in recent posts here, here, here, and here, are why your SBM bloggers do what we do. Biologists, other scientists, and intellectuals in general have joined the battle against the pseudoscientific travesty known as “intelligent design.” Many physicians, however, even of the brainy, academic variety, act as though the equally pseudoscientific but more dangerous travesty known as “integrative medicine” is either a good thing or, at least, is a necessary addition to medical school curricula.
The Case of Yale Medical School and David Katz
Yale Medical School provides a disturbing example. It has a new ”integrative medicine” program that touts the usual advertising slogans, including the vision of ”an integrative healthcare system that combines the best evidence-based therapeutics from both conventional and non-conventional medical systems.” It has joined the Consortium of Academic Health Centers for Integrative Medicine, a beneficiary of the wealthy Bravewell Collaborative, whose commitment to mandating quackery in all of medicine has been discussed several times on SBM. Some of the “people” in the Yale program are the conspicuous woo-meister Bernie Siegel; the lesser-known woo-meister David Katz, Director of the Yale Prevention and Research Center; and a naturopath who graduated from Bastyr University.
In April, the program hosted the 1st Annual Yale Integrative Medicine Scientific Symposium. After a full day of pablum and nonsense, the symposium came in for some well-deserved criticism, especially here, here, here, and here. It was directed mostly at two participants: Yale’s Dean of Medical Education, Richard Belitsky, whose obsequious welcoming statement had betrayed either an ignorance of science and critical thinking or an ignorance of “CAM”; and David Katz. Katz had given a talk in which he presented several disconfirming trials of IMCs and concluded that “it’s an invitation to think more fluidly about evidence.” Curiously, he had previously authored a book titled Clinical Epidemiology & Evidence-Based Medicine: Fundamental Principles of Clinical Reasoning & Research.
Dr. Katz, who can serve up tu quoque and PoMo drivel with the best of them, responded to his detractors with the “medicine doesn’t know everything” ploy:
I recently gave a talk on evidence-based holistic care at the Yale School of Medicine’s inaugural conference dedicated to ‘integrative medicine,’ generally defined as the blend of conventional and alternative medical practices. This is the model we use at the Integrative Medicine Center I run in Derby, CT (www.imc-griffin.org). I was subsequently branded a quack in certain quadrants of the blogosphere. So I am writing a defense. But not of myself, nor of my talk, nor of Yale, nor the conference. But of patients.
…
The view I expressed, and that guides our practice, is that human need goes on long after the results of randomized clinical trials start to run thin. I do not think doctor and patient should part company there. I believe that responsible use of the science we have should not preclude responsiveness to the needs of patients that fail to respond as textbooks say they should to that all-too-limited science. I believe, in other words, that patient need, not trial results, should be the ultimate master medical care must serve.
I learned devotion to evidence-based practice from my teachers and professional colleagues; I was pushed toward integrative medicine by the needs of my patients.
…
Some, apparently, see quackery in a willingness to try to help a patient feel better past the point where scientific evidence is decisively clear. But does it actually make sense to subordinate patient need to the limits of what we know with certainty, when what we strive to know through biomedical research is all in an effort to satisfy patient need? If so, I suppose I would rather be a duck than a doctor, so claims of quackery are just so much water off my back.
By Sleight of Pen: Language Distortion is Power
Many readers of that passage, even those in academic medicine, might fail to recognize its subtle language distortions. “Trial results,” “science,” and “what we know with certainty” are casually presented as interchangeable terms, with the tacit presumption that none would disagree. It follows that the limits of science are found when “the results of randomized clinical trials start to run thin.” Of course, “what we know with certainty” is “all-too-limited.” Who would disagree with that? The toolbox of a practitioner constrained by ”science” must be very small indeed.
It also follows that in the much larger realm of practices outside “that all-too-limited science,” there is no basis for distinguishing one from another. Since science has nothing to say about them, all are equally promising. All are thus fair game for the practitioner “pushed toward integrative medicine by the needs of [his] patients.”
In clinical situations where trial results do not provide clear guidance, the only alternative to “alternative medicine,” says Dr. Katz, is to abandon the patient. Such a ”when did you stop beating your wife” argument is insulting to all honest physicians, who either know or ought to learn how to deal with such situations without sacrificing their professional integrity.
What, exactly, is in store for Dr. Katz’s patients when “the results of randomized clinical trials start to run thin”? Obligingly, he directs us to the website of his Integrative Medicine Center at Griffin Hospital in Derby, CT. In addition to Dr. Katz, who refers to himself as a “nationally renowned authority,” the Integrative Medicine Center Team includes two naturopaths, a nurse practitioner who specializes in Therapeutic Touch, and a young MD from Pakistan who, having won a gold medal for being the “top medical graduate” from Baqai University, derailed an apparently promising career by selecting Griffin Hospital for his internal medicine residency.
If it Walks Like a Duck and Quacks Like a Duck…
“Treatment approaches available at the IMC include”:
internal medicine, naturopathic medicine, preventive medicine, nutritional counseling, nutritional supplements, nutriceuticals, herbal medicine, acupuncture, craniosacral therapy, Therapeutic Touch, homeopathy, intravenous micronutrient therapy (Myers’ Cocktail), relaxation therapies, as well as referrals to counselors, trauma therapists (EMDR), and chiropractic.
The links above, which I added, demonstrate that contrary to Dr. Katz’s assertions there is plenty of science—trials or no trials—pertaining to the corresponding methods, and that science convincingly demonstrates each to be bogus. The most obvious example is homeopathy, which is as valid as a perpetual motion machine and just as deserving of a trial. “Myers’ Cocktail,” an intravenous infusion of calcium, magnesium, and several vitamins, has all the characteristics of snake oil. There is no reason to predict that it would benefit anyone not suffering from a frank deficiency of one or more of its ingredients, the combination of which is arbitrary. All are easily absorbed through the GI tract, thus raising the question of why they are given intravenously, which is both more dangerous and more expensive than a pill.
The obvious answers are two: the “placebo effect” is enhanced by the invasive procedure, and it is more lucrative for the quack. The Myers’ Cocktail champion, Alan Gaby, a former faculty member at Bastyr University, promotes it as a near panacea:
Conditions that frequently responded included asthma attacks, acute migraines, fatigue (including chronic fatigue syndrome), fibromyalgia, acute muscle spasm, upper respiratory tract infections, chronic sinusitis, and seasonal allergic rhinitis. A small number of patients with congestive heart failure, angina, chronic urticaria, hyperthyroidism, dysmenorrhea, or other conditions were also treated with the Myers’ and most showed marked improvement. Many relatively healthy patients chose to receive periodic injections because it enhanced their overall well being for periods of a week to several months.
If Dr. Katz lacks the street savvy to recognize the concoction for what it is, he might consider the results of his own trial of the Myers’ Cocktail for patients with fibromyalgia. Although his 2006 abstract tortures the data to suggest efficacy, none was found. Since the treatment is implausible in the first place, this amounts to pretty solid evidence against it. Nevertheless, nearly two years later Dr. Katz still peddles Myers’ Cocktail at the IMC, or so we gather from the website. He must have “thought more fluidly about the evidence.”
Thus by distorting language in order to bait and switch, Dr. Katz and Yale University have transformed “responsiveness to the needs of patients that fail to respond…to that all-too-limited science” into that all-too-familiar, all-too-unlimited quackery. Abracadabra!

   Science, Reason, Ethics, and Modern Medicine, Part 3: Implausible Claims and Formal Ethics Statements
Kimball Atwood
Link
The Ethics of Implausible Medical Claims (IMC)
In Part 2 of this series* we learned from David Katz, MD, a key member of the Yale School of Medicine’s “integrative medicine” program, that he had been “pushed toward integrative medicine by the needs of [his] patients.” We also learned that Dr. Katz’s rationale for this decision justifies a wide range of quackery—both in principle and in fact. I had previously alluded to arguments like those of Dr. Katz in a comment on SBM several months ago:
…we must be true to medical ethics, no matter what else we do. If that means losing a few patients, so be it. Patients are free agents, and we can only do so much to influence them. To the extent that we don’t do that as well as we might (which is obviously true in some cases), we might do better. But our ethical obligation is to science and truth; it is not, as many modern physicians would have it and as much as we may lament sometimes losing patients to woo, to seducing patients to stick with us no matter what, if the “what” includes engaging in a charade about “integration” or “complementary therapies”…
Realizing that some might argue that physicians’ obligations to patients ought to trump their obligations to “science and truth,” I later revised that statement:
Several weeks ago I argued here that a physician’s primary ethical obligation is to science and truth. In retrospect I probably should have put it a slightly different way: a physician’s primary ethical obligation is the same as everyone else’s. It is to honesty and integrity. For physicians, however, that means being true to real medical knowledge, among other things, and real medical knowledge comes from science.
In spite of that revision, two readers whose opinions I respect challenged my assertion. Dr. Peter Moran’s worthy efforts to educate patients about the realities of “alternative” cancer treatments are considerable. Here on SBM he has repeatedly challenged us to explain how, when confronted with testimonials of “alternative” cures, we ought to respond without using “a high-handed, ‘we know best’ stance” and thus “appear to want to distance [ourselves] from the intimate concerns of [our] patients.” I was thinking mainly of him when I wrote the revision above, because on this key topic—how to respond ethically, but with compassion, to patients who want to believe in implausible treatments—I’ve come to think of Dr. Moran as the “conscience” of Science-Based Medicine. Those with cancer, he has reminded us, “are folk very like you and me who are simply grasping at any straw that might save or prolong their lives.” His take on why IMCs are appealing to those with less ominous problems is well-developed and agrees with my own, mostly. We part ways, however, when he concludes (also here and here) that ethical physicians might have good reasons—unlike Dr. Katz’s—to entertain benign, if implausible treatments:
I thought KA was similarly slightly off-target when defining the primary ethical obligations of the physician as being to science without even mentioning the patient. That would certainly apply in an ICU or in the operating theatre, and in most specialist neurological or oncological practices (hint!), but there may be quite sound reasons why many family doctors don’t yet want to completely abandon dubious but relatively safe methods in favour of the contents of their drug cabinet.
Co-blogger David Gorski also took issue with my assertion about honesty and integrity/science and reason:
…ethically as a physician, my first obligation is to the patient and seeing that the patient is treated as effectively as she can be (sorry, Kimball, my first obligation when treating a patient is not to science or reason). Telling a patient too stridently that her dubious therapy is, well, a dubious therapy risks doing one of two things: (1) driving her away from effective science-based therapy or (2) making her reluctant to tell me everything she is taking.
Dr. Gorski and I, however, don’t really disagree. The relevant terms in his passage and in my second passage quoted above are “primary,” “first,” and “too stridently.” I have no trouble with Dr. Gorski’s description of how he responds to patients who ask about or elect dubious treatments:
As long as they tell me about it and it doesn’t interfere with scientific-based treatment, patients can do whatever they like. If they ask my my opinion, I will tell them, as non-judgmentally as possible, but I’m not going to push too hard.
My original assertion about medical ethics, quoted above, did not argue that our obligation to science and reason trumps our obligation to our patients. It argued that our obligation to science and truth (reason is also fine) trumps “seducing patients to stick with us no matter what, if the ‘what’ includes engaging in a charade [see Dr. Katz] about ‘integration’ or ‘complementary therapies’…” Later I used the term “primary ethical obligation,” which was unfortunate because the word “primary” distracted from the real point: Honesty and integrity, which in medicine must not clash with science and reason, are necessary for fulfilling our ethical obligations to our patients.
It is beside the point to quibble over whether, on some numbered list, our obligation to our patients should precede or follow our obligation to honesty and integrity. We can’t fulfill our obligation to our patients if we don’t fulfill our obligation to honesty and integrity, and we can’t do that if we overlook science and reason.
Formal Treatises
Most modern statements of medical ethics include both an obligation to patients and an obligation to honesty and integrity, which in turn is either explicitly or implicitly linked to science. The recent “Medical Professionalism in the New Millennium: A Physician Charter,” a joint project of the American College of Physicians and the European Federation of Internal Medicine, has three Fundamental Principles, the first two of which are:
Principle of primacy of patient welfare. This principle is based on a dedication to serving the interest of the patient. Altruism contributes to the trust that is central to the physician–patient relationship. Market forces, societal pressures, and administrative exigencies must not compromise this principle.
Principle of patient autonomy. Physicians must have respect for patient autonomy. Physicians must be honest with their patients and empower them to make informed decisions about their treatment. Patients’ decisions about their care must be paramount, as long as those decisions are in keeping with ethical practice and do not lead to demands for inappropriate care.
There is also a Set of Professional Responsibilities, among which are:
Commitment to professional competence. Physicians must be committed to lifelong learning and be responsible for maintaining the medical knowledge and clinical and team skills necessary for the provision of quality care. More broadly, the profession as a whole must strive to see that all of its members are competent and must ensure that appropriate mechanisms are available for physicians to accomplish this goal.
Commitment to honesty with patients. Physicians must ensure that patients are completely and honestly informed before the patient has consented to treatment and after treatment has occurred. This expectation does not mean that patients should be involved in every minute decision about medical care; rather, they must be empowered to decide on the course of therapy.
Commitment to scientific knowledge. Much of medicine’s contract with society is based on the integrity and appropriate use of scientific knowledge and technology. Physicians have a duty to uphold scientific standards, to promote research, and to create new knowledge and ensure its appropriate use. The profession is responsible for the integrity of this knowledge, which is based on scientific evidence and physician experience.
Among the more venerable AMA Principles of Medical Ethics are these:
I. A physician shall be dedicated to providing competent medical care, with compassion and respect for human dignity and rights.
II. A physician shall uphold the standards of professionalism, be honest in all professional interactions, and strive to report physicians deficient in character or competence, or engaging in fraud or deception, to appropriate entities.
V. A physician shall continue to study, apply, and advance scientific knowledge, maintain a commitment to medical education, make relevant information available to patients, colleagues, and the public, obtain consultation, and use the talents of other health professionals when indicated.
VIII. A physician shall, while caring for a patient, regard responsibility to the patient as paramount.
The AMA code also has several relevant policies:
E-3.01 Nonscientific Practitioners
It is unethical to engage in or to aid and abet in treatment which has no scientific basis and is dangerous, is calculated to deceive the patient by giving false hope, or which may cause the patient to delay in seeking proper care.
E-3.04 Referral of Patients
A physician may refer a patient for diagnostic or therapeutic services to another physician, limited practitioner, or any other provider of health care services permitted by law to furnish such services, whenever he or she believes that this may benefit the patient. As in the case of referrals to physician-specialists, referrals to limited practitioners should be based on their individual competence and ability to perform the services needed by the patient. A physician should not so refer a patient unless the physician is confident that the services provided on referral will be performed competently and in accordance with accepted scientific standards and legal requirements.
E-8.20 Invalid Medical Treatment
The following general guidelines are offered to serve physicians when they are called upon to decide among treatments: (1) Treatments which have no medical indication and offer no possible benefit to the patient should not be used (Opinion 2.035, “Futile Care”). (2) Treatments which have been determined scientifically to be invalid should not be used (Opinion 3.01, “Nonscientific Practitioners”).
E-9.132 Health Care Fraud and Abuse
The following guidelines encourage physicians to play a key role in identifying and preventing fraud: (1) Physicians must renew their commitment to Principle II of the American Medical Association’s Principles of Medical Ethics which states that “a physician shall deal honestly with patients and colleagues, and strive to expose those physicians deficient in character, competence, or who engage in fraud or deception.” (2) Physicians should make no intentional misrepresentations to increase the level of payment they receive or to secure non-covered health benefits for their patients.
H-480.963 Folk Remedies Among Ethnic Subgroups
The AMA: (1) does not recommend the sole use of unvalidated folk remedies to treat disease without scientific evidence regarding their safety or efficacy; (2) encourages research to determine the safety and efficacy of folk remedies; (3) physicians should be aware that the use of folk remedies may delay patients from seeking medical attention or receiving conventional therapies with proven benefit for disease treatment and prevention; (4) practicing physicians should routinely ask patients whether they are using folk medicine or family remedies for their symptoms. Physicians can educate patients about the level of scientific information available about the therapy they are using, as well as conventional therapies that are known to be safe and efficacious; and (5) physicians should be aware of folk remedies in use and the level of scientific information available about such remedies, and should include this information when discussing conventional treatments and therapies with their patients.
H-480.964 Alternative Medicine
Policy of the AMA on alternative medicine is: (1) There is little evidence to confirm the safety or efficacy of most alternative therapies. Much of the information currently known about these therapies makes it clear that many have not been shown to be efficacious. Well-designed, stringently controlled research should be done to evaluate the efficacy of alternative therapies. (2) Physicians should routinely inquire about the use of alternative or unconventional therapy by their patients, and educate themselves and their patients about the state of scientific knowledge with regard to alternative therapy that may be used or contemplated. (3) Patients who choose alternative therapies should be educated as to the hazards that might result from postponing or stopping conventional medical treatment.
H-480.983 Clinical Ecology
It is the policy of the AMA (1) to continue to monitor the published literature on clinical ecology and to report as appropriate; and (2) that those who support a new test, procedure, or treatment must prove by appropriately controlled peer reviewed trials that it is effective for the purposes for which it is used and that the burden should not be shifted to opponents to prove that a new test or therapy is invalid.
E-8.085 Placebo Use in Clinical Practice.
A placebo is a substance provided to a patient that the physician believes has no specific pharmacological effect upon the condition being treated. In the clinical setting, the use of a placebo without the patient’s knowledge may undermine trust, compromise the patient-physician relationship, and result in medical harm to the patient.
Physicians may use placebos for diagnosis or treatment only if the patient is informed of and agrees to its use. A placebo may still be effective if the patient knows it will be used but cannot identify it and does not know the precise timing of its use. A physician should enlist the patient’s cooperation by explaining that a better understanding of the medical condition could be achieved by evaluating the effects of different medications, including the placebo. The physician need neither identify the placebo nor seek specific consent before its administration. In this way, the physician respects the patient’s autonomy and fosters a trusting relationship, while the patient still may benefit from the placebo effect.
A placebo must not be given merely to mollify a difficult patient, because doing so serves the convenience of the physician more than it promotes the patient’s welfare. Physicians can avoid using a placebo, yet produce a placebo-like effect through the skillful use of reassurance and encouragement. In this way, the physician builds respect and trust, promotes the patient-physician relationship, and improves health outcomes (I, II, VIII).
Other pertinent entries among the AMA ethics policies are Withholding Information from Patients (Therapeutic Privilege) and several statements about Health Fraud.
Although the language could have been more definitive, it seems to me that the statements above deem it unethical for physicians to offer implausible treatments, to refer patients to others for implausible treatments, or, if asked, to fail to inform patients of the implausible nature of such treatments. They also deem it unethical to administer a placebo without the patient’s informed consent, or to mislead patients about the reasons that implausible treatments make some people feel better. Thus it is dishonest to recommend acupuncture or homeopathy in a disguised attempt to elicit a placebo effect.
I will not argue that physicians must, perforce, honor certain written edicts. On the contrary, if these treatises contained language condoning the cited practices, I would reject them. My reason for citing the statements above is not to pose as a petty, medical ethics traffic cop. It is to remind us that these ethical precepts have been considered at length by thoughtful, committed physician-ethicists, have been around long enough to have made their way into formal opinions, and have persisted in those opinions. Thus physicians practicing in disregard for them—especially those, such as Dr. Katz and other IMC enthusiasts, who have been flouting them for several years—should be aware that they have chosen to repudiate the considered opinions of wise men and women, regarding an issue that is fundamental to medical ethics: the relation of honesty and integrity to science and reason.
The medical profession as a whole ought to acknowledge that as well. The time has come for the profession to revisit those ethical precepts and decide whether they have become obsolete. If not, it’s time to clean house.

   Science, Reason, Ethics, and Modern Medicine Part 5: Tu Quoque and History
Kimball Atwood
Link
My Discussion with Dr. P
After last week’s post, Dr. Peter Moran answered with more salient points. I’ll spend this week discussing those, because I share Dr. Moran’s “interest in examining the kind of messages we are putting out.” Acknowledging the inequality inherent in his not being the blog author, I’ll offer the last word to Dr. Moran by ending this series* and letting whatever comments he may have in response to today’s post be the last, at least for now.
Here is Dr. Moran’s response to my response:
The standard healthfraud position is that even if CAM may sometimes be helpful to people, it is a bad thing regarding which there can be no compromise.
It is a bad thing because, to quote KA, “implausible claims being promoted, tacitly or otherwise” leads to “unnecessary expense, a patient later feeling betrayed by her physician when the treatment doesn’t work or the ‘placebo effect’ wears off and she realizes that it was only a ‘pretend medicine’ all along, other patients or physicians concluding that the treatments have validity (and possibly being convinced to forgo rational treatment), people in general losing confidence in modern medicine when it appears that many physicians can’t distinguish between the rational and the bogus, wasted research funds, human subjects unnecessarily duped and endangered, a befuddled media, a befuddled public, befuddled leglislators endorsing quacks to a befuddled public,” [and more].
Superficially this is a powerful line of argument, because we can fairly easily find instances where such predictions are fulfilled. Yet I have suggested that healthfrauders routinely exaggerate the risks of CAM, that some potential dangers are not realized much in practice, and it again needs to be said that the medical risks depend very much upon the specific “quack” claim. Also, some of these outcomes, if as truly awful as implied, may be thought of as mainly exposing a puzzling lack of reach for more rational voices. This is the very reason why I have an interest in examining the kind of messages we are putting out.
And there is considerable evidence against some of the above. Surveys routinely show that the majority of CAM-using patients express great satisfaction with them. Despite this, nearly all also use conventional methods, and are satisfied with that. Even in this heyday of CAM, very few of the public use CAM exclusively.
The Scots are an interesting case study. They don’t seem to be vastly worse off health-wise or science-wise, or even exceptionally befuddled, yet 50% of their doctors prescribe homeopathic remedies and a further 10% use herbs.
What this adds up to is that the public may not be (generally) as indiscriminate in their use of alternative methods and as in need of protection from unscientific concepts as we healthfrauders like to think. Maybe we skeptics encounter a disproportionate amount of the dangerous and nutty extremes of “alternative” medicine because we go looking for it and we hang about on forums that gather it. We are also fairly obviously more deeply offended by it than most, which can lead to confirmation bias.
Which brings me to this: in science-based medicine we are quite accustomed to evaluating medical options in terms of benefits vs. risks. We normally try to quantify them so that we can reach a reasonable balance. An example is studies designed to determine the optimum dose of a noxious chemotherapeutic agent, so that cancer remissions are produced at acceptable risk. These days such studies even often include quality-adjusted life year (QALY) measurements so that all aspects of patient life can bear upon a precise, measurement-based, final judgment.
That’s how we normally operate. Not so, apparently, when it comes to CAM. In relation to that we adopt the meanest, least flexible possible position, even while lacking any clear evidence-based handle on the medical or psychosocial benefits or the social functions of unconventional and folk medical activity, or their risks. All we know, having completely dismissed all the anecdotal evidence, is that they usually don’t work obviously better than placebo when put to the test in the banal environment of the typical controlled trial. We don’t care that they may be being used mainly when there is no entirely satisfactory evidence-based treatments for the complaint, nor that some of the methods may simply have not yet been subjected to sufficient study to be quite sure whether they have intrinsic medical activity or not. It’s all one, if it is seen to be “alternative”.
Is the resort to ethical argument about the use of placebo medicines a lazy way of justifying such a globally rigid approach? The ethical card has some force, even when set against other primary ethical obligations of doctors towards their patients, but in reality it can only apply to that select few who have a clear enough grasp of the scientific evidence to be as certain as can be that any given CAM treatment IS a placebo. But Healthfraud has to also deal with what a lot of other practitioners think and what our patients are often trying out on their own initiative. It is a waste of time talking ethics to those who don’t share quite the same scientific convictions, and that applies to almost everyone.
KA seems to help my argument with personal medical experiences that are reminiscent of some of stories of “alternative” testimonial. I was quite struck by them, for I have had very similar personal experiences and observed them in many patients. They illustrate how there can be a complete disconnection between the benefits of medical interactions and their scientific content, and, indeed, any treatment content.
We have no basis on which to assume that similar benefits are not regularly occurring within CAM, especially when a lot of people are telling us so. In fact, KA elsewhere is arguing the reverse, that while CAM may be offering placebo and other non-specific medical benefits these can equally well occur within the 5-15 minute hurly-burly of the typical present day medical practice, I am sure they can, with good doctors, but I would love to be quite sure about the “equally”. Depending on the condition, and with as much reluctance and regret as anyone here, my money would be on the practitioner who is not hamstrung by what science says, and in consequence has a confident explanation for everything and never runs out of remedies. Sure, some of his clients may eventually realize that he is actually a nutcase, but even most of those will probably find ways in which they think he has helped them.
The reason I am being a bit stubborn about these matters is that I have what is as yet merely a vague inkling that they are important — that they may, despite initial appearances, help rather than damage our purposes.
Those who we might wish to influence in various ways may not have sufficient familiarity with science and medicine to be able to articulate the line of argument I am developing (it is difficult enough for me and I know I have a long haul ahead of me even if I am entirely right), but one of the reasons for our distressing inability to have the influence we desire may well be that people of ordinary intelligence and experience can nevertheless almost instinctively sense the weaknesses and exaggerations in some of our habitual arguments. They switch off. We may be able to do better with a change of tack that might actually be more in touch with medical realities.
Be reassured that looking at CAM methods in strict cost/risk/benefit terms will definitely not change most of our judgments concerning CAM. The main difference would be in the emphasis of argument and some of the rhetoric. We would still be able to describe Hulda Clark as a dangerous and severely deluded cancer quack, or, if you prefer, an unspeakable criminal fraud.
My Response to His Response to My Response
Let me say, again, that I am not trying to prohibit free citizens from pursuing whatever “CAM” they feel like pursuing. What I’m against is physicians violating their own ethical mandates, and governments and medical schools tacitly or explicitly promoting pseudomedicine. If 50% of Scots use homeopathy, so be it, even if it illustrates a deficit of science education in Scotland (I agree that “some of these outcomes…may be thought of as mainly exposing a puzzling lack of reach for more rational voices.” We’ve discussed that before: it argues for a better reach for rational voices, not for rational voices condoning the irrational). If “50% of [Scottish] doctors prescribe homeopathic remedies,” that is a different and real problem, both for medical ethics and for the regulation of physicians in Scotland. If Scots are not ”vastly worse off health-wise,” that is a triumph of modern public health and scientific medicine in spite of homeopathy, not because of it. Scots are obviously worse off “science-wise” for homeopathy, whether or not this has been measured, just as Americans would be worse off science-wise if “intelligent design” advocates had their way.
Yes, “we can fairly easily find instances where such [dire] predictions are fulfilled.” We on SBM have written about many of them. How many such instances does it take to counteract the (possibly) many more instances of patients expressing “great satisfaction” with “CAM”? Very few, it seems to me. I think that the public is in need of protection from unscientific concepts, both because some people are not as capable of discriminating as others, and for the larger societal reasons listed above. Even if that were not the case, my point in these posts is that physicians ought not to be dishonest. The border between what a physician can and cannot ethically recommend must be the point at which honesty and integrity are compromised, not the point at which some arbitrary distinction is made between “safe” and unsafe implausible treatments—even if pseudoscience might make some patients feel better.
A Digression
A regular column in the New York Times Magazine is “The Ethicist,” by Randy Cohen. Readers send real examples of ethical dilemmas for commentary by Mr. Cohen, who is very good at providing it. The July 27th issue included a non-medical illustration that “little white lies” are not the best policy:
The Ethicist
Hidden Opinions
By RANDY COHEN
Published: July 27, 2008
A friend, a nonwriter, retired after a distinguished career, asked me to read his screenplay because I am a published author. It was terrible: poorly written with no plot. Worse, it was clearly autobiographical, abounding in self-aggrandizement and intimate sexual details. To be kind, I told him merely that it was too personal. Recently he met a literary agent. I fear he will share this document with the agent and be humiliated. Must I now tell him my full opinion? — NAME WITHHELD, ARIZONA
If we rejected every work of fiction marred by vanity or cringe-worthy self-exposure, miles of shelf space would be left vacant at Barnes & Noble. And as a commercial matter, it is curious to discourage your friend from writing about himself in an era when memoirs proliferate like kudzu, which had sex with some other kudzu, to which it was not married. But it is in neither your literary judgment nor your commercial savvy that you most failed your friend — and fail him you did — but in your lack of candor. If he requested your honest opinion (and not just praise, another possibility), you should have provided it, as gently as possible. To be misleading is not to be kind.
It’s no picnic telling a friend that you don’t think much of his work, but you agreed to give him your professional assessment, and that’s what you should have done…
To be fair, I make the following disclosure. I asked Mr. Cohen’s permission to include that quotation in this blog, briefly suggesting to him how I would use it. He kindly gave his permission, but with this caveat:
I’m not sure I agree that the editor-writer relationship is analogous to the doctor-patient relationship, and I should also mention that many people—many writers—disagree with me here. Several said they’d absolutely lie to a non-writer friend who sought their opinion; they’d cosset the non-writer with praise.
I think that Mr. Cohen should give himself more credit. If anything, the doctor-patient relationship requires more ethical rigor, as it were, than does an informal interaction in the “outside world.” What many writers say they would do, moreover, is only barely relevant to the ethics of the situation. I agree that “to be misleading is not to be kind,” and also that there is a gentle, constructive way to be honest in such situations. That applies to writers talking to non-writers and to physicians talking to patients.
Digression Complete
Back to Dr. Moran. I’m not sure I understand his point about “quality-adjusted life year (QALY) measurements.” How can chemotherapy, a rational treatment whose effects, both good and bad, are well documented, be compared to “balancing qi”? We know a lot about implausible treatments. We know that they are implausible. We need to know about their “social functions” as much or as little as we need to know about the social functions of our patients’ religious beliefs. Sometimes that may be considerable (as in Dr. Moran’s previous example of the witch doctor), other times not at all, but never does it mean that we ought to promote such social functions—any more than we would promote religious beliefs.
“CAM” is not necessarily “all one, if it is seen to be ‘alternative’.” The occasional “CAM” treatments that are plausible—some molecules, mainly—deserve the same consideration from physicians as do plausible new drugs that have not yet been approved for general use: we should favor appropriate investigations but not premature recommendations, except in extraordinary circumstances. Other treatments that are typically labeled “CAM” but are not, such as relaxation, exercise, some diets, and some manual therapies, are perfectly rational and are not at issue here.
Was Dr. Moran referring to physicians when he wrote of the “select few who have a clear enough grasp of the scientific evidence…”? If so, that is a sad commentary on recent medical training. It also may reflect a little-appreciated deficiency of Evidence-Based Medicine, i.e., its failure to acknowledge the necessity of scientific plausibility. Regarding the charge that to “resort to ethical argument about the use of placebo medicines” is “lazy,” I think that in the current cultural environment, the easier and therefore lazier stance for physicians is to acquiesce to the “CAM” fad (just as to “cosset the non-writer with praise” is the lazier stance for writers). This is evidenced by the widespread encroachment of “CAM” onto medical school campuses and the almost complete lack of formal opposition.
Regarding the “placebo effects” that I reported having experienced (my “personal medical experiences”), I don’t see how those help Dr. Moran’s argument. The whole point of those experiences was that they occurred in the context of rational medicine. I don’t see any “disconnection between the benefits of medical interactions and their scientific content” in those experiences. On the contrary, it was the rational content of the information that I got from those interactions that made me feel better. Even if other people might get similar comfort from irrational content, so what? I have not denied that in any of these discussions; I have argued that physicians ought not to steer patients in that direction by subterfuge or ignorance, and that rational medicine has a similar, if sometimes unrealized capacity for comforting patients.
Dr. Moran’s next paragraph is the most troublesome. He surely knows that impugning the “5-15 minute hurly-burly of the typical present day medical practice” can only be an argument for changing the way present day medicine is practiced, not an argument for “CAM.” Regarding non-specific medical benefits that might accrue from “the practitioner who is not hamstrung by what science says, and in consequence has a confident explanation for everything and never runs out of remedies,” what is new or worthy about that? It’s a perfect description of the classical quack. It is remiscent of a passage quoted previously on SBM, whose very authors appeared to have missed the point:
Chiropractic finds its voice exactly where biomedicine becomes inarticulate. Too often, biomedicine fails to affirm a patient’s chronic pain. Patients think their experience is brushed aside by a physician who treats it as unjustified, unfounded, or annoying, attitudes that heighten a patient’s anguish and intensify suffering. Chiropractors never have to put a patient’s pain in the category of the “mind.” They never fail to find a problem. By rooting pain in a clear physical cause, chiropractic validates the patient’s experience.
Should MDs aspire to similar, deceptive practices, merely because they “validate the patient’s experience”? I’m sure Dr. Moran will agree that the passage does not constitute an argument for that. All it does is portray chiropractors as sleazy salesmen.
Conclusion
If rational medical practitioners need “a change of tack that might actually be more in touch with medical realities,” such a change might include offering patients more time, better explanations, more honesty, more sympathy, and a more creative range of rational treatments. All of Dr. Moran’s points argue for those. None argues for “CAM.”

   Ann Coulter says: Radiation is good for you!
David Gorski
Link
Here at Science-Based Medicine, we try to be relatively apolitical. We might not always succeed, but in general our main concern is not so much with right-wing or left-wing politics, but rather with how prevailing government policies and regulation impact the delivery of medical care, in particular whether they tend to prevent, do nothing about, or promote the proliferation of non-science-based medical care. Consequently, when Kimball or I call for the disbanding of the National Center for Complementary and Alternative Medicine (NCCAM), it does not matter one whit to us who is President or which party controls Congress. All that matters is that we see NCCAM as a government entity that, through credulously studying many “alternative medicine” modalities, ends up inadvertently promoting them and providing them with the imprimatur of government approval. The same concept applies to state medical licensing boards licensing pseudoscientific modalities, such as naturopathy, acupuncture, and homeopathy. By regulating these “disciplines,” states also provide them with an unmerited mantle of respectability through their imprimatur of regulating them as professions, just like medicine and nursing.
As far as political views, although all of us have them and they occasionally even come to the fore in disagreements (remember when Wally Sampson occasionally clashed with others with differing political viewpoints?), we generally subsume them for purposes of the SBM blog experience into our advocacy for basing medicine on the best science available. Sometimes, however, when a pundit or politician makes claims that are either contrary to or distort science for ideological or political advantage, I feel the need to discuss those claims, sometimes even sarcastically. Such was the case last week, when Ann Coulter wrote a blisteringly ignorant column, entitled A Glowing Report on Radiation. She wrote this article in the wake of the fears arising in Japan and around the world of nuclear catastrophe due to the damage to the Fukushima nuclear power plant caused by the earthquake and tsunami that hit northern Japan on March 11. Coulter was subsequently interviewed by Fox News pundit Bill O’Reilly on The O’Reilly Factor on Thursday evening:
Yes, according to Coulter, radiation is good for you, just like toxic sludge! Even more amazing, in this video Bill O’Reilly actually comes across as the voice of reason, at least in comparison to Ann Coulter. He’s very skeptical of Coulter’s claims and even challenges her by saying, “So by your account we should all be heading towards the nuclear reactor.”
So, fellow SBM aficionados, is Coulter right? Are all those scientists warning about the dangers of even low-level radiation all wrong? Should we start hanging out in radioactive mine shafts, as Coulter mentions in her column (seriously) in order to boost our health and decrease our risk of cancer?
Not so fast, there, Ann. Here’s a hint: If Bill O’Reilly can lecture you on science and look more reasonable than you, you’re off the rails.
Coulter, hormesis, and “Don’t worry, be happy!”
Actually, the scientific assessment of what levels of exposure to ionizing radiation are dangerous is, as you might imagine, a wee bit more complicated than my little sarcastic rejoinder makes it, but you’d never know that from Ann Coulter’s article and her interview with Bill O’Reilly. The reason for my sarcastic characterization of Coulter’s scientific nonsense is because her article uses many of the same tactics as any denialist. Chief among these is that Coulter takes the germ of a scientific controversy and then uses it to try to imply that the scientific consensus is fatally flawed. In this case, the scientific controversy is over how dangerous low level exposure to radiation is used to imply that the radiation from a nuclear disaster is not potentially harmful. All you former residents of Chernobyl, take note! It’s fine to move back to your homes that you were forced to abandon 25 years ago!
Here is what Coulter claims in her article:
With the terrible earthquake and resulting tsunami that have devastated Japan, the only good news is that anyone exposed to excess radiation from the nuclear power plants is now probably much less likely to get cancer.
This only seems counterintuitive because of media hysteria for the past 20 years trying to convince Americans that radiation at any dose is bad. There is, however, burgeoning evidence that excess radiation operates as a sort of cancer vaccine.
As The New York Times science section reported in 2001, an increasing number of scientists believe that at some level — much higher than the minimums set by the U.S. government — radiation is good for you. “They theorize,” the Times said, that “these doses protect against cancer by activating cells’ natural defense mechanisms.”
What Coulter is referring to is the phenomenon of radiation hormesis. This is nothing more than a biphasic dose-response curve to radiation in which the curve initially goes down with increasing dose (less risk of disease with increasing radiation exposure) and then curves upward and at some point crosses a threshold where radiation exposure is no longer beneficial but harmful with further dose increases. Basically, it’s a scientific model wherein low level exposure to radiation is not only not harmful but in fact beneficial. The reason for this effect, if it exists in humans, is hypothesized to be that low level radiation activates DNA damage repair and other protective mechanisms that are not activated in the absence of radiation; moreover, it is further hypothesized that these mechanisms are activated more than they need to be, so that low level radiation is actually protective against radiation-induced diseases such as cancer.
The radiation hormesis model is markedly different from the currently prevailing model that is used for regulatory purposes by most governments, the linear no-threshold (LNT) model, which states that there is no such thing as a “safe” dose of radiation and that radiation dammage accumulates in a linear fashion with dose. For completeness sake, I will note that there is also at least one other model for the biological effects of radiation, specifically a model in which there is a threshold dose under which radiation is not harmful. In practice, distinguishing between a threshold model and a hormesis model can be very difficult.
In order to give you an idea of what hormesis would look like in a radiation dose-response curve, I stole this graph from Wikipedia. Actually, I didn’t steal it; it’s public domain because it’s a product of a U.S. government agency. However, it illustrates the concept of hormesis quite well:
Curve A demonstrates supralinearity, in which toxic effects are actually more intense per unit of radiation at lower doses; there is no evidence that this is indeed the case. Curve B is linear, and Curve C is linear-quadratic, in which low doses of radiation are less harmful per unit of radiation than higher doses. Curve D represents hormesis, where low doses of radiation are actually protective up to a certain threshold, where the curve shifts from a protective effect to a harmful effect with increasing radiation. The main contenders for the model that best describes radiation effects are either curve B, C, or D.
The key aspect of Coulter’s article that makes it so irresponsible is what she leaves out. What she neglects to mention is that, even if hormesis is an accurate model for radiation effects in humans, it only applies for very low dose exposures. (More on how low in the next section.) True, Coulter does at one point concede that it is “hardly a settled scientific fact that excess radiation is a health benefit,” throughout the rest of her article she presents the idea of hormesis as though it were–you guessed it!–a settled scientific fact. Indeed, Coulter’s earlier assertion that “excess radiation acts as sort of a cancer vaccine” is the sheerest exaggeration, even if hormesis is an accurate model of radiation exposure. Aside from this major exaggeration, how do Coulter’s assertions, which appear to be based largely on studies cited in a single NYT article that is nearly a decade old, stack up against science?
Not very well. Surprise! Surprise! As is the case with many denialists, Coulter takes a germ of actual science and then twists and exaggerates it beyond all recognition in order to support a preconceived notion, namely that those pointy-headed (and, of course, liberal) environmentalists are hiding the evidence that radiation at low doses is good for you. To accomplish this, Coulter cherry picks studies, failing to put them into their proper context with existing research, all for the purpose of advancing her ideological viewpoint.
Radiation hormesis: Ann Coulter’s claims versus reality
Before I discuss what the data regarding radiation hormesis actually show, it’s essential to discuss briefly why it is that the LNT model predominates when it comes to policy-making and setting limits on what is considered “safe” radiation exposure. The reason is not that biased scientists are “hiding” the evidence that radiation is good for you. Rather, it boils down to a few reasons. The first is probably that an LNT model is the simplest, most conservative model that can be fit to currently existing evidence. The problem with the LNT model is the same as the problem with the hormesis model. While at higher radiation doses, effects due to radiation are, like effects due to pretty much any other high-level environmental exposure, much more robust and reproducible, at lower radiation doses, the effects are weaker, and the scatter in the data is much greater. In other words, at low doses the signal-to-noise ratio is much lower due to a lot more “noise” and a lot less signal in the data. Moreover, the data are more difficult to collect, and variability from system to system, organism to organism, and cancer to cancer is likely to be much greater.
As imperfect as it is, the LNT model is a reasonable approximation for purposes of policy-making because it is conservative and safe. Admittedly, there are problems applying such a model when the doses get really low, as in lower than the normal background radiation that we all live in, but it’s a useful approximation. When it is very hard to distinguish between an LNT model and a hormesis model at very low radiation exposures, until better data can be gathered that clearly demonstrate the superiority of one model over another, the responsible and safe model to choose is the most conservative one that fits reasonably well. Basing public policy on a model that, if incorrect, has the potential to result in considerable harm in the form of increased radiation-induced disease prevalence is not wise policy at all, at least when the alternate model is not demonstrably wrong.
As far as Coulter’s reliance on an old NYT article, I thought I’d take a look at the article itself. As an aside, I can’t help but note that I really hate it when the online version of an article doesn’t include links to cited articles, and Coulter is no different in this regard. However, I do believe I managed to find this 2001 NYT article anyway from November 27, 2001, entitled For Radiation, How Much Is Too Much? It’s by Gina Kolata and discusses the controversy that had begun to bubble up about what doses of ionizing radiation might be considered safe. If you read it, you’ll see that it’s much more balanced than how it is portrayed by Coulter. For example, here is what Coulter writes about two studies cited by Kolata:
Among the studies mentioned by the Times was one in Canada finding that tuberculosis patients subjected to multiple chest X-rays had much lower rates of breast cancer than the general population.
Here is what Kolata actually wrote about these studies:
Now, some scientists even say low radiation doses may be beneficial. They theorize that these doses protect against cancer by activating cells’ natural defense mechanisms. As evidence, they cite studies, like one in Canada of tuberculosis patients who had multiple chest X-rays and one of nuclear workers in the United States. The tuberculosis patients, some analyses said, had fewer cases of breast cancer than would be expected and the nuclear workers had a lower mortality rate than would be expected.
Dr. Boice said these studies were flawed by statistical pitfalls, and when a committee of the National Council on Radiation Protection and Measurement evaluated this and other studies on beneficial effects, it was not convinced. The group, headed by Dr. Upton of New Jersey, wrote that the data “do not exclude” the hypothesis. But, it added, “the prevailing evidence has generally been interpreted as insufficient to support this view.”
Notice how the finding in “some analyses” that there were fewer cases of breast cancer than might be expected has magically morphed into “tuberculosis patients subjected to multiple chest X-rays had much lower rates of breast cancer than the general population” in Coulter’s words. Also note that this appears to be the NCRPM report that analyzed the data. Unfortunately, it would have cost me $40 to download the PDF; so I didn’t. But what about these studies?
The first study to which Coulter refers appears to be a study from Canada that was reported in the New England Journal of Medicine in 1989. This study examined the mortality from breast cancer in a cohort of 31,710 women who had been treated for tuberculosis at Canadian sanatoriums between 1930 and 1952. A significant proportion (26.4%) of these women had received radiation doses to the breast of 10 cGy or more from repeated fluoroscopic examinations during therapeutic pneumothoraxes. It should be noted that these sorts of doses of radiation are far in excess of anything likely to be received using modern radiological equipment, in particular given that we no longer perform fluoroscopy and therapeutic pneumothorax to treat tuberculosis. Interestingly, this is how the abstract summarizes the results of this study:
Women exposed to ≥ 10 cGy of radiation had a relative risk of death from breast cancer of 1.36, as compared with those exposed to less than 10 cGy (95 percent confidence interval, 1.11 to 1.67; P = 0.001). The data were most consistent with a linear dose–response relation. The risk was greatest among women who had been exposed to radiation when they were between 10 and 14 years of age; they had a relative risk of 4.5 per gray, and an additive risk of 6.1 per 104 person-years per gray. With increasing age at first exposure, there was substantially less excess risk, and the radiation effect appeared to peak approximately 25 to 34 years after the first exposure. Our additive model for lifetime risk predicts that exposure to 1 cGy at the age of 40 increases the number of deaths from breast cancer by 42 per million women.
Oops! Maybe I found the wrong study! On the other hand, this is a Canadian study that looked at women with tuberculosis who received numerous chest X-rays (fluoroscopy, actually), and I can’t find another one like it. I also couldn’t find other publications with other analyses. The analysis that exists in the published literature, for better or for worse, concludes that the risk of breast cancer is elevated with exposures to radiation greater than 10 cGy. So, what are these other “analyses” that purport to claim that these patients actually had a lower risk of mortality from breast cancer? I smelled a rat.
My first hint came from an article published in the Journal of the Association of American Physicians and Surgeons (JPANDS) by Bernard Cohen entitled The Cancer Risk From Low Level Radiation: A Review of Recent Evidence. I’ve discussed JPANDS and how it plays fast and loose with science for ideological reasons before, in particular its antivaccine views and its publishing studies so bad that laughter is the only appropriate response. In his article, Cohen claims that hormesis “found for breast cancer among Canadian women exposed over longer periods of time to X-ray fluoroscopic examinations for tuberculosis (13); when appropriately evaluated, this evidence shows a decrease in risk with increasing radiation dose at least up to 20 cSv (20 rem).” Unfortunately, no evaluation of this evidence is included; Cohen simply asserts that this is so.
Fortunately, it didn’t take long for me to find other JPANDS articles making the same argument. For example, this one by Joel M. Kauffman. In it, Kaufmann divides up the subjects into several radiation dose ranges, while rejecting data from Nova Scotia because “too few” low radiation points were included. Conveniently he fails to define what “too few” is. However, if one looks at Table I in the NEJM paper, it’s obvious that in the dose range between 10 and 99 cSv, the death rate in Nova Scotia was much higher than the other provinces. One wonders if that had anything to do with leaving out the data, rather than writing the authors for a more detailed breakdown of the data between those dose levels, one does. In any case, what Kaufmann appears to have done is what JPANDS writers frequently do: Cherry pick the data. He took the lower end of the dose ranges, used “eyeball” fitting instead of statistical fitting to models, and left out any hint of a statistical analysis. The authors of the NEJM article went to great lengths to demonstrate that a LNT model was the best fit to their data; Kaufmann expects you to “eyeball” his graph and accept his claim of hormesis. Similarly, Jerry Cuttler and Myron Pollycove, in another JPANDS article, plotted the Canadian data on a semilog scale to make a hormesis effect look far more convincing than the actual data support, all the while simply claiming that a hormesis model fit the data better than an LNT model. Unfortunately, they didn’t “show their work,” so to speak. No discussion of how they modeled the data is included. No wonder the NCRPM found these “other” analyses unconvincing. Also, while it’s not surprising that Coulter would have gotten her data on this from JPANDS, it’s rather disappointing that Kolata didn’t look deeper back in 2001.
The second study cited by Kolata and exaggerated by Coulter was a study of U.S. nuclear industry workers. Regarding this sort of data, the scientists at the Lawrence Berkley National Laboratory have included on their website this analysis:
The results of individual studies have been inconclusive, and to investigate the matter further a combined analysis has been carried out of seven studies–three for sites in the United States (Hanford, Oak Ridge, and Rocky Flats), three for sites in the United Kingdom, and one for Canada. A total of 95,673 workers was included, of whom 60% received effective doses above 10 mSv (1 rem). In the entire population, there were 15,825 deaths, of which 3,976 were from cancer. The comprehensive results for all cancers taken together showed a very slight decrease in cancer rate with increasing dose. However, this result had no statistical significance. Of possible greater statistical significance is a slight increase with radiation dose for some types of leukemia. Overall, the statistical uncertainties were large enough that the analysis did not rule out linearity or any of the other alternative dose-response curves indicated in Figure 15-1–although it does set an upper limit on the possible magnitude of a hypothesized supra-linearity effect.
The study being discussed it this one, which, by the way, concludes:
These estimates, which did not differ significantly across cohorts or between men and women, are the most comprehensive and precise direct estimates of cancer risk associated with low-dose protracted exposures obtained to date. Although they are lower than the linear estimates obtained from studies of atomic bomb survivors, they are compatible with a range of possibilities, from a reduction of risk at low doses, to risks twice those on which current radiation protection recommendations are based. Overall, the results of this study do not suggest that current radiation risk estimates for cancer at low levels of exposure are appreciably in error.
Coulter also makes much of a study of shipyard workers from 1991:
A $10 million Department of Energy study from 1991 examined 10 years of epidemiological research by the Johns Hopkins School of Public Health on 700,000 shipyard workers, some of whom had been exposed to 10 times more radiation than the others from their work on the ships’ nuclear reactors. The workers exposed to excess radiation had a 24 percent lower death rate and a 25 percent lower cancer mortality than the non-irradiated workers.
The reference for this is:
Matanoski, G. M. (1991) Health Effects of Low-Level radiation in Shipyard Workers, Final Report, DOE/EV/10095-T2, National Technical Information Service, Springfield, Virginia, USA.
Unfortunately, I couldn’t get a hold of this report online over the weekend. I did, however, find the more recent reanalysis of the data from 2008 by Matanoski et al published in the Journal of Radiation Research. What Matanoski found wa that most of the differences in mortality and cancer rates found between shipyard workers who serviced nuclear ships and shipyard workers who did not were not significant, although there did appear to be trends towards increased risk of leukemias and other cancers with increasing dose. Overall, as far as saying anything about the association between radiation exposure and cancer, at best this study could be described as inconclusive. Certainly it’s exceedingly thin gruel to make such definitive statements about hormesis. As for the lower all-cause mortality among the nuclear workers, that is almost certainly due to phenomenon known as the “healthy worker effect“; i.e., the selective recruiting of healthier than average persons into the industry who have continued access to better than average health care.
Similarly thin gruel is this claim by Coulter:
In 1983, a series of apartment buildings in Taiwan were accidentally constructed with massive amounts of cobalt 60, a radioactive substance. After 16 years, the buildings’ 10,000 occupants developed only five cases of cancer. The cancer rate for the same age group in the general Taiwanese population over that time period predicted 170 cancers.
The people in those buildings had been exposed to radiation nearly five times the maximum “safe” level according to the U.S. government. But they ended up with a cancer rate 96 percent lower than the general population.
Not exactly. Actually, not at all. It’s not even thin gruel; it’s misrepresentation, either intentional or through Coulter’s laziness in researching the article. Coulter, as usual, is exhibiting willful ignorance by citing old data. In fact, more recent analyses of the Taiwanese population that lived in these buildings do not support her claims at all. The most recent followup study I could find was published in 2006 in the International Journal of Radiation Biology by Hwang et al. The results were:
A total of 7271 people were registered as the exposed population, with 101,560 person-years at risk. The average excess cumulative exposure was approximately 47.8 mSv (range 5 1 – 2,363 mSv). A total of 141 exposed subjects with various cancers were observed, while 95 developed leukemia or solid cancers after more than 2 or 10 years initial residence in contaminated buildings respectively. The SIR were significantly higher for all leukemia except chronic lymphocytic leukemia (n1⁄46, SIR1⁄43.6, 95% confidence interval [CI] 1.2–7.4) in men, and marginally significant for thyroid cancers (n1⁄46, SIR 1⁄4 2.6, 95% CI 1.0 – 5.7) in women. On the other hand, all cancers combined, all solid cancers combined were shown to exhibit significant exposure-dependent increased risks in individuals with the initial exposure before the age of 30, but not beyond this age.
Hwang et al concluded:
The results suggest that prolonged low dose-rate radiation exposure appeared to increase risks of developing certain cancers in specific subgroups of this population in Taiwan.
So, basically, Coulter is completely wrong about the Taiwan incident. There is an increased incidence of cancer in young people, at least, who lived in those apartment buildings. Science is hard, isn’t it? Coulter’s also on seriously dubious footing when she cites Professor Bernard L. Cohen, whose various studies of the relationship between radon and lung cancer buck the established consensus that radon is a risk factor for lung cancer. (Yes, this is the very same Bernard Cohen who wrote the JPANDS article I mentioned earlier in this post; to me his having published in JPANDS is to me a huge hit on any credibility he might have had.) It turns out that Cohen probably didn’t control adequately for smoking in his studies because a reanalysis of his reported data demonstrated similar, strongly negative correlations between radon exposure and cancers strongly linked to cigarette smoking and weaker negative correlations between radon and cancers moderately associated with smoking. No such correlation was found for cancers not linked to smoking. These results strongly suggest that Cohen didn’t adequately control for smoking in his analysis. Another criticism points out that Cohen fell prey to the ecological fallacy and suggested that county-level data probably do not represent the best units to detect a correlation between radon and lung cancer.
Coulter’s final claims center on the Chernobyl disaster and victims of the atomic bombings of Hiroshima and Nagasaki. In particular, she claims that only 30 people died in the plant as a direct result of the disaster and further downplays the risk of cancer in the survivors, stating:
Even the thyroid cancers in people who lived near the reactor were attributed to low iodine in the Russian diet — and consequently had no effect on the cancer rate.
As is usually the case for any scientific claims made by Coulter, this is utter rubbish. Unfortunately for Coulter, her timing in publishing her article was exquisitely bad. On the very next day after her article was published, the National Cancer Institute released the most comprehensive study yet of thyroid cancer in Chernobyl survivors. The findings indicated that radioactive iodine (131I) from the fallout from the reactor was likely responsible for thyroid cancers that are still occurring among people who lived near the reactor and that the risk of this cancer is not declining. In other words, no, Ann, the hugely elevated levels of thyroid cancer among people who live near Chernobyl when the reactor disaster occurred are not due to iodine deficiency in the Russian diet. There is some evidence that iodine deficiency might have increased the risk of 131I-induced cancers, particularly in the youngest, but that’s not what Coulter said. She implied that iodine deficiency could account for the elevated incidence of thyroid cancer among those affected by the fallout. Much more about the health effects of the Chernobyl disaster can be found here. It should also be noted that most people who lived in the area were not exposed to that much radiation according to the United Nations-sponsored team investigating. Most were exposed to about 9 mSv, about 1/3 the equivalent of a CT scan of the chest, abdomen, and pelvis, once the short-term doses to the thyroid were subtracted
Poor Ann. That’s what you get for not doing a bit more research. Basically, every claim she makes in her article can be shown to be either mistaken, grossly exaggerated, or based on old evidence. She even cites Tom Bethell, author of The Politically Incorrect Guide to Science, as a source. Bethell is an all-purpose right-wing science denialist, who, besides viewing scientists as attention whores who trump up alarmist findings in order to secure more research funding and castigates science for its commitment to “materialism,” also denies evolution and anthropogenic global warming. He even rejects relativity and embraces “AIDS reappraisal,” while extending his view on hormesis to argue that hormesis actually protects us from toxic chemicals in the environment that, according to him, we don’t have to worry about nearly as much as environmentalists say we do. In fact, Coulter includes a paragraph in her article that is so unintentionally hilarious that I can’t help but cite it:
Although it is hardly a settled scientific fact that excess radiation is a health benefit, there’s certainly evidence that it decreases the risk of some cancers — and there are plenty of scientists willing to say so. But Jenny McCarthy’s vaccine theories get more press than Harvard physics professors’ studies on the potential benefits of radiation. (And they say conservatives are anti-science!)
I doubt that Coulter appreciates the irony encompassed by this paragraph, given that this paragraph is further encompassed by an article that uses many of the same deceptive techniques of argumentation that the anti-vaccine movement, as epitomized by Jenny McCarthy, likes to use. She then digs herself in deeper by correctly mentioning that Botox is a poison that is safe to use at high doses (Jenny McCarthy loves Botox, actually) and then pointing out the principle that many poisons are safe and beneficial at low doses but dangerous at high doses. If these arguments didn’t occur within the context of her spewing of misinformation, Coulter might actually be making some sense. Too bad she couldn’t resist adding:
Every day Americans pop multivitamins containing trace amount of zinc, magnesium, selenium, copper, manganese, chromium, molybdenum, nickel, boron — all poisons.
They get flu shots.
Perhaps Coulter has more in common with Jenny McCarthy than she would like to admit. Actually, there’s no “perhaps” about it. Coulter will also say whatever fits her political viewpoint. Last week she was ranting about how radiation is good for you. Back in November, she was complaining that the new 
These scanners result in a dose of 0.001 mSv for about 5 seconds of full body exposure, and even frequent fliers would be exposed to much less radiation than Coulter is claiming to be just fine. Indeed, Ann Coulter should be lining up to be scanned. After all, that little radiation is good for you!
Is hormesis a real phenomenon?
Despite my irritation, I was rather grateful for Coulter’s article. It did remind me of a rather fascinating debate in radiobiology over what model best describes the biological effects of radiation. Hormesis might indeed be a real phenomenon in humans, but it’s been very difficult to demonstrate. Even one of the best review articles I’ve found that argues for the existence of hormesis as a phenomenon, an article by Tubiana et al entitled The Linear No-Threshold Relationship Is Inconsistent with Radiation Biologic and Experimental Data doesn’t exactly argue for hormesis. Rather, it argues that the LNT model is inconsistent with the data and needs to be modified to more of a threshold model, in which doses below a certain threshold are probably harmless but above a certain threshold start to increase the risk of disease. Arrayed against these sorts of arguments are scientists like Rudi H. Nussbaum and Wolfgang Köhnlein, who call hormesis and the zero-risk threshold dose “scientifically refuted, but stubborn myths.” They even argue that in some cases the risk of low level radiation exposure might well be underestimated. Not surprisingly, in her article Coulter used nearly every myth that Nussbaum and Köhnlein deconstruct in their paper.
Hormesis is clearly an area of science that is as yet controversial. The reason is because it’s difficult to demonstrate definitively one way or another whether hormesis occurs in humans in response to low dose radiation. As I mentioned above, the signal-to-noise ratio for studies of low dose radiation is very low. Moreover, studies of low dose radiation have been conflicting, although we can say with a fair amount of confidence, based on my review of the literature, that, if hormesis occurs, it probably occurs only below doses of 100 mSv. Remember, 30 mSv is the dose received from a CT scan of the chest, abdomen, and pelvis and can be estimated to increase one’s lifetime risk of a fatal cancer by 1 in 1000 to 1 in 500 in pediatric patients, while most people receive around 3 mSv per year from background radiation. To put this all into context, XKCD has a very useful chart that describes how much radiation we receive from various sources. Another good perspective comes from a recent AP article on the topic, which takes a much more balanced perspective.
The bottom line is that we just don’t know whether hormesis is a real phenomenon for radiation response in humans. Lacking that knowledge, we do know that the LNT model is a reasonable approximation for purposes of regulation because it is simple and defensible. Even so, different professional organization bodies have started to question it. For example, the French Academy of Sciences and National Academy of Medicine published a report in 2005 that stated:
In conclusion, this report raises doubts on the validity of using LNT for evaluating the carcinogenic risk of low doses (< 100 mSv) and even more for very low doses (< 10 mSv). The LNT concept can be a useful pragmatic tool for assessing rules in radioprotection for doses above 10 mSv; however since it is not based on biological concepts of our current knowledge, it should not be used without precaution for assessing by extrapolation the risks associated with low and even more so, with very low doses (< 10 mSv), especially for benefit-risk assessments imposed on radiologists by the European directive 97-43.
The Health Physics Society’s position statement, revised in July 2010, states:
In accordance with current knowledge of radiation health risks, the Health Physics Society recommends against quantitative estimation of health risks below an individual dose of 5 rem in one year or a lifetime dose of 10 rem above that received from natural sources. Doses from natural background radiation in the United States average about 0.3 rem per year. A dose of 5 rem will be accumulated in the first 17 years of life and about 25 rem in a lifetime of 80 years. Estimation of health risk associated with radiation doses that are of similar magnitude as those received from natural sources should be strictly qualitative and encompass a range of hypothetical health outcomes, including the possibility of no adverse health effects at such low levels.
Again, we just don’t know. My guess is that hormesis, if it occurs in humans in response to radiation, is not nearly as potent a phenomenon as its adherents claim. My further guess is that the way hormesis is invoked as a scientific explanation for homeopathy doesn’t help its reputation. Be that as it may, until science settles the question, I do know that, contrary to what Coulter claims in her nonsensical arguments, low dose radiation is not a magical “cancer vaccine.” At the very best, low dose radiation might not hurt you or might have some very slight benefits. At worst, it might actually hurt you more than the current scientific consensus accepts. That’s too wide range of possibilities and too much uncertainty to be laying down a barrage of misinformation as intense as Coulter’s.
ADDENDUM: Here’s an amusing little takedown of Coulter’s nonsense, for your edification. I had a good chuckle at this comment by Gordon Bloyer, who writes:
Schultz telling anyone about science, LOL. Coulter wrote an extensive column using back-up from science. Sgt. Ed should learn to read.
Coulter didn’t give her “theory” she cited science. She is right and O’Reilly just shoots off his mouth before he lets others complete a sentence. In this case he had NO IDEA what Coulter was talking about.
I invite Gordon to read my discussion of hormesis and see that the “back-up from science” that Coulter used is anything but. I do thank Gordon, though, for a moment of hilarity in a painful day of grant writing. Ditto felixw, who comments:
I know that O’Reilly went to Harvard. And Ann Coulter graduated cum laude from Cornell, where she founded The Cornell Review before getting her law degree at the University of Michigan, where she edited the law review. Then she clerked for the United States Court of Appeals for the Eighth Circuit.
Given how much a law school education has to do with science, by this logic, I should be able to confidently and definitively make pronouncements on the law! After all, I graduated from the University of Michigan, too, just like Ann Coulter! :-) 

   Surprise, surprise! Dr. Andrew Weil doesn’t like evidence-based medicine
David Gorski
Link
Dr. Andrew Weil is a rock star in the “complementary and alternative medicine” (CAM) and “integrative medicine” (IM) movement. Indeed, it can be persuasively argued that he is one of its founders, at least a founder of the its most modern iteration, and I am hard-pressed to think of anyone who did more in the early days of the CAM/IM movement, back before it ever managed to achieve a modicum of unearned respectability, to popularize CAM. In fact, no physician that I can think of has over the course of his lifetime done more to promote the rise of quackademic medicine than Dr. Weil. The only forces greater than Dr. Weil in promoting the infiltration of pseudoscience into academic medicine have been the Bravewell Collaborative and the National Center for Complementary and Alternative Medicine (NCCAM). Before there was Dr. Mehmet Oz, Dr. Dean Ornish, Dr. Mark Hyman, or any of the other promoters of IM, there was Dr. Weil.
And why not? Dr. Weil looks like an aging 1960s rock star, and, operating from his redoubt at the University of Arizona, is quite charismatic. For all the world he has the appearance of a kindly, benevolent Arizona desert Santa Claus, an ex-hippie turned respectable dispensing advice about “natural” medicines, writing books, and making himself ubiquitous on television and radio whenever the topic of alternative medicine comes up. Before Dr. Oz told Steve Novella that “Western” science and medicine can’t study woo like acupuncture, Dr. Weil was there, paving the way for such arguments, previously considered ludicrous, to achieve a patina of respectability.
In fact, he’s still at it, doing it far better and far more subtly than the ham-handed Dr. Oz. Unfortunately, it’s the same anti-science message and the same appeal to other ways of knowing built upon tearing down straw men versions of evidence-based medicine (EBM) with gusto. This was brought home last week when Dr. Weil co-authored an opinion piece with Drs. Scott Shannon and Bonnie J. Kaplan for the journal Alternative and Complementary Therapies entitled Safety and Patient Preferences, Not Just Effectiveness, Should Guide Medical Treatment Decisions, an article that was noted at the blog Booster Shots in a credulous, fawning post entitled Dr. Weil says there’s a better approach to evaluating clinical drug trials. In contast, Steve Novella put it far more succinctly (and accurately) in the title of his post: Andrew Weil Attacks EBM. That’s exactly what Weil and company did in this article.
While Steve is absolutely correct, I also see it more as Dr. Weil demonstrating once again that, upstarts like Dr. Oz aside, he is still the master of CAM/IM apologia, much as, even though both were Sith Lords, Emperor Palpatine remained master over Darth Vader until just before the end. You’ll see why in terms of the arguments, both subtle and not-so-subtle, that Dr. Weil and his acolytes make. Moreover, even though his disciple Shannon is granted the coveted first author position, the arguments presented leave little doubt that it’s Weil who’s driving the bus.
Efficacy, effectiveness, and safety, oh, my!
Shannon et al begin by emphasizing that there is a difference between “efficacy” and “effectiveness,” and from the very beginning take pains to separate their definitions of the two, pointing out that they will not use the terms interchangeably. Specifically, they refer to the definition of “efficacy” as “demonstration of benefit under ideal conditions, typically in randomized controlled trials” and “effectiveness” as “demonstration of benefit in real-life conditions.” These are relatively new concepts, comparatively speaking. Even so, they are nothing “alternative,” being instead firmly within the bailiwick of EBM. This doesn’t stop Shannon et al from presenting these concepts as though EBM and science-based medicine (SBM) don’t consider questions of whether treatments work under real-life conditions. As you will see, it is also the opening volley in a rather complex set of fallacious arguments designed to support the watering down of scientific rigor in order to allow CAM/IM modalities that can’t stand up under close scientific investigations to be labeled as “effective” (or at least acceptable).
First of all, let’s see a bit more what Weil and company are up to when they compare “effectiveness” with “efficacy.” Mostly, it’s meant to attack the reliance of EBM randomized clinical trials (RCTs) in order to set the stage for proposing something else more to their liking:
The value of efficacy lies mainly in its ability to indicate potential for effectiveness accurately. Sadly, in the drive to emphasize the importance of delineating clearly sound measures of clinical effectiveness, modern medicine has come to equate RCTs as the final arbitrators of clinical decision making. As discussed below, RCTs are but one tool to sort out these complex questions. In integrative medicine (IM), particularly, with its emphasis on patient variables and practitioner participation, evaluation of efficacy is not sufficient.
I agree that one major value of efficacy in RCTs is that it indicates the potential for effectiveness in the real world. I also even partially agree that EBM overemphasizes RCTs as the final arbiters of clinical decision-making, although, I note, not for the same reason as Weil and his acolytes. We at SBM frequently make the argument that plausibility based on science should be a much more important consideration in evaluating treatments than it is in EBM. Indeed, this is a major reason that the term science-based medicine, as opposed to evidence-based medicine, was coined in the first place. EBM emphasizes RCTs without considerations of prior plausibility, relegating basic science considerations that conclusively demonstrate CAM/IM modalities like homeopathy to be so implausible as to be not worth studying in humans to its lowest rung in its scale of evidence. So, while we agree somewhat that EBM overemphasizes RCTs, we most definitely do not agree with Dr. Weil about how it does so. Whereas Dr. Weil seems to think that RCTs are overemphasized because they are too standardized and do not adequately take into account real-world conditions, he most definitely would not like SBM’s plea to EBM to consider scientific plausibility, because doing so would relegate modalities like energy healing, homeopathy, and acupuncture to the hopelessly implausible.
Dr. Weil also does a very clever thing with his argument. While he correctly points out that evaluation of efficacy is not sufficient to determine whether a treatment is effective, as Steve Novella pointed out, Dr. Weil conveniently neglects to acknowledge that it is at the very least necessary. If RCTs indicate that a therapy is not detectably more efficacious than a placebo, which is the case for the vast majority of CAM/IM modalities, then there is no point in doing trials in “real world” conditions because there is no reason to expect the to be effective. Basically, RCTs serve as a screening test to identify promising therapies that are likely to be effective in real world use. In essence, Dr. Weil is echoing the same two-pronged attack that CAM/IM advocates make against RCTs, the first being the claim that “Western science” can’t study his woo, which is the same fallacious argument that Dr. Oz made regarding acupuncture. The second prong of the attack is to point to what Weil calls “alternative sources of valid information” as demonstrating that his woo works when RCTs do not support its efficacy. Although Dr. Weil is careful never to state this explicitly or even use the term “pragmatic trial,” his emphasis on “effectiveness” over “efficacy” is in essence a plea for relying much more heavily on observational trials, in particular pragmatic trials.
But what are “pragmatic” trials? They are trials that seek to determine if a treatment is effective outside the confines of an RCT, in other words, out there in the “real world.” RCTs, because they seek to determine efficacy, need to control for as many potentially confounding factors as possible, which makes them inherently artificial to one extent or another. Once efficacy is established under controlled conditions, it is sometimes then useful to determine whether this efficacious treatment is effective under usual circumstances in the “real world,” where patient compliance might be poor, either due to side effects, cost, or difficulty in adhering to the therapy; where complicated protocols might be more problematic to follow compared to academic medical centers; and where use of the treatment will inevitably be expanded to patient populations not represented in the RCTs used to approve the drug.
Pragmatic trials can go one of two ways. In the case of treatments demonstrated efficacious in RCTs, most commonly pragmatic trials demonstrate a lower level of effectiveness than the efficacy measured in RCTs might indicate. As Harriet Hall discussed a couple of years ago, a classic example of a treatment that was shown to be efficacious in RCTs but potentially more dangerous in pragmatic studies of actual use in community hospitals was the clot-busting drug t-PA, which was effective for ischemic strokes in RCTs but resulted in a higher death rate. For efficacious interventions, the “real world” is almost always less hospitable than the “ideal world” of RCTs. There is one case, however, where the “real world” can make a treatment seem more effective than it is by any objective measure, and that’s for treatments that are essentially placebos. Outside of the rigorous, carefully controlled world of RCTs, placebo medicine can seem to have effectiveness, which is exactly why proponents of pseudoscientific medicine love pragmatic trials. The populations aren’t as well-defined; often there is no placebo control; and they are almost always unblinded. There is a good reason why, after well-designed RCTs have demonstrated acupuncture to be no more efficacious than placebo or sham acupuncture, acupuncturists have returned to pragmatic studies, as, I will note in a shameless plug, Harriet Hall will discuss tomorrow. The hospitableness of pragmatic trials to placebo medicine is also behind much of the attack on RCTs by advocates of pseudoscience, and Shannon et al certainly engage in such attacks with gusto:
RCTs have dominated decision making about efficacy in health care for almost 50 years. Many researchers have explored the difficulty of subjecting IM treatment approaches to RCTs. There are some characteristics of IM interventions that make RCTs particularly difficult to carry out, and perhaps even less relevant, than for conventional allopathic medicine. As Fønnebø pointed out, the gap between published studies of integrative approaches on the one hand, and the clinical reports by practitioners on the other hand, may partially result from the fact that placebo-controlled RCTs are designed to evaluate pharmaceutical interventions.
Or, more likely, they result from the biases and lack of adequate controls for placebo effects inherent in “clinical reports” by practitioners. Also note the Oz-like argument that RCTs can’t evaluate Weil’s favored woo.
Granted, uncontrolled clinical reports from practitioners can, if carefully documented, represent one form of pragmatic trials; more often they are at best preliminary data and at worst nothing more than anecdotes. To support their attack, Shannon et al cite a prominent New England Journal of Medicine study, namely Concato et al, which found that well-designed observational studies often produce the same results as RCTs, meaning that placing observational studies lower on the rung of the EBM hierarchy might not be justified. Unfortunately, the authors failed to note that Concato et al only looked at studies of clinical questions with objective outcomes, including the Bacille Calmette-Guérin vaccine and tuberculosis, mammography and mortality from breast cancer, cholesterol levels and death due to trauma, treatment of hypertension and stroke, and treatment of hypertension and coronary heart disease. Moreover, Concato et al have been criticized for cherry picking their examples. In contrast, most CAM modalities are designed to treat symptoms with a major subjective component. Also going against Weil’s argument that, because RCTs are designed for pharmaceutical treatments, they might not be so good evaluating non-pharmaceutical treatments is a more recent study comparing effects in RCTs versus observational studies in digestive surgery, which found that a quarter of observational studies gave different results than randomized trials and that between-study heterogeneity was more common in observational studies. If this study holds up, it would appear that RCTs work just fine (and better than observational studies) for surgical interventions.
The merits and flaws of Concato et al aside, it is a false dichotomy that we must choose either controlled RCTs or or less rigorously controlled observational studies. EBM and SBM encompass them all; they simply disagree on how much relative weight to give each type of evidence. Fortunately, Weil doesn’t make the argument that we must favor one or the other. Unfortunately, he’s more subtle than that. What he does instead is to argue for decreased influence of RCTs by arguing that they should be only one part of the picture, an argument he makes by bringing up the Bradford-Hill criteria.
Trying to shrink the importance of the RCT
The heart of the argument in this paper is that RCTs should be only one (presumably of many) factors that determine whether medicine considers and intervention to be effective or not. As the title suggests, safety and patient preferences play a prominent role in Weil’s argument, but first Weil yokes poor Sir Austin Bradford-Hill, FRS, a pioneering early to mid-20th century epidemiologist best known for his work linking smoking to lung cancer and pioneering the RCT, into his service. Bradford-Hill proposed a set of criteria for drawing causality about disease etiology, but his nine criteria are sometimes used in considering the effectiveness of treatments.
Shannon et al set the stage:
As he [Bradford-Hill] pointed out, not all criteria are appropriate for all issues being analyzed, but he listed nine in total from which appropriate ones should be selected for any given situation:
    • Strength—referring to the robustness of the association between the causative agent and the outcome
    • Consistency—meaning being able to obtain similar results across different research sites and methodologies (i.e., replication)
    • Specificity—by which Bradford-Hill meant one disease having one specific outcome, which may not be relevant to complex disorders (e.g., psychiatric problems)
    • Temporality—referring to the commonsense notion that the cause always precedes the outcome
    • Biologic gradient—which is best described as a dose– response curve: increased treatment would presumably result in a proportionate increase in the effect (again, not relevant in all disorders)
    • Plausibility—referring to whether the results are biologically sound
    • Coherence—which refers to the agreement of a study’s findings with what is already known (hence, not relevant in situations of truly novel interventions)
    • Experiment—the situation in which randomly introducing the causative agent results in the outcome
        ◦ Analogy—which is the idea that a similar cause results in a similar outcome.
It is particularly interesting to note that the Bradford-Hill criteria, specified by the individual who influenced the methodology we now accept for RCTs massively, lists experiment with randomization methods as only one of nine criteria for establishing causality.
See, you nasty, reductionist scientists? RCTs are only one criteria! They’re not the be-all and end-all of clinical evidence! I can’t help but point out that there is an implicit straw man here in that embedded in many of the Bradford-Hill criteria are the same sorts of arguments we make at SBM, particularly criteria numbers 1, 2, 5, 6, and 7. In particular, we at SBM like criteria numbers 6 and 7. Key to the very concept of SBM is that interventions should have biological plausibility. They should also be congruent with what is known about the disease or, if not congruent, the evidence supporting an intervention must be sufficiently compelling that it justifies overthrowing the existing paradigm (for example, in the case of the discovery that H. pylori causes peptic ulcers). I also can’t help but point out that most CAM/IM treatments still fail most of Bradford-Hill’s criteria. In particular, CAM/IM treatments almost always run afoul of all of Bradford-Hill’s criteria other than #4, and, quite frankly, sometimes the wackier ones even seems as though they could run afoul of #4 as well.
Perhaps realizing this, Weil says nothing more about Bernard-Hill criteria other than to mention them again briefly in his conclusion. Instead, he and his merry trio of woo-apologists move on, Gish gallop-like, to other deficiencies in the “Western, reductionistic” model.
Use and abuse of systems biology and “holistic” methods
Unable to abuse poor Bradford-Hill anymore, Weil moves on to list “unique” features of CAM/IM research that—surprise!—turn out to be not-so-unique and not nearly as difficult to take into account in SBM (or even in EBM) as he implies:
For instance, the healing relationship of a doctor and patient is generally excluded or “controlled for” in conventional RCTs, whereas some researchers would argue that unconditional positive regard forms the underpinnings of the healing relationship between two people.
The enhanced focus on the healing relationship is thus another factor delineating IM from conventional health care models. A second example is the concept of individualized care, which is rarely included in RCTs (perhaps the MTA study in childhood ADHD is a notable exception). The notion that each patient is unique and quite different permeates IM.
I like the admission that most CAM/IM is placebo medicine, though. Oh, you didn’t see that? The emphasis on the “healing relationship” between practitioner and patient tells you that what Weil is talking about is placebo medicine. After all, that relationship is very important to placebo effects.
Be that as it may, I’ve discussed the so-called “individualization” of CAM treatments before, as well as the difference between the “personalized medicine” when practitioners of SBM use the term versus when CAM/IM advocates use the term. For the full discussion click on the links; the CliffsNotes version follows. Basically, the notion that each patient is unique is a notion that is recognized in EBM and SBM. It’s also utterly facile and obvious, given that no two people are alike. What EBM and SBM try to do is to classify and stratify patients based on science-based characteristics that can be objectively related to disease severity, etiology, and response to therapy. As genomic medicine, systems biology, proteomics, and metabolomics become more sophisticated, it has become possible to make finer and finer distinctions between patients based on biology. The hope is that ultimately knowing these fine distinctions will allow us to “personalize” therapies to individual patients, thus ushering in an era of truly personalized medicine.
In contrast, in CAM/IM the idea of “individualization” and “personalized” medicine most frequently boils down to making it up as the practitioner goes along and doing whatever the practitioner feels like, all without a basis in sound science and evidence. In fact, Weil appeals to just such “personalization” or “individualization” based on magic and fairy dust as being a problem with applying RCTs to CAM/IM:
However, classic RCT research design requires patients to be broken out into groups with a similar diagnosis, which impairs the ability to evaluate an individualized treatment system, such as classical homeopathy, Traditional Chinese Medicine, or Ayurveda. In each of these systems, the patient must be individualized into a quite unique pattern that does not lend itself to a more broad disease generalization such as that found in conventional allopathic medicine. Curiously, the cutting edge of modern medicine anticipates that customized and individualized care looms as a result of advances in single nucleotide polymorphisms (SNPs) and the ability to create a specific genetic fingerprint for each individual.
Imagine that! Using groups that are based on diagnosis and actual disease biology, rather than based on prescientific vitalistic thinking! The nerve of those reductionistic “Western” physicians!
Here’s a hint: Genome variations, as demonstrated by SNPs, are not an example of the sorts of classifications one finds in homeopathy, traditional Chinese medicine, or Ayurveda, although I am impressed at the attempt to liken CAM classifications to “cutting edge” genomic variations and science-based classification systems. Sadly, though, leaving aside the utter ridiculousness of the analogy, Weil appears to be behind the times when it comes to personalized medicine and genomics. Ask any systems biologist or geneticist. SNPs aren’t even really “cutting edge” anymore. Maybe they were ten years ago (possibly even five years ago), but other methods and markers are rapidly supplanting SNPs, in particular direct sequencing of relevant parts of the genome, a technique made practical by next generation sequencing technologies and the concomitant exponential plummeting of the cost of such analyses. Also, as Harriet Hall has pointed out, the claim to “personalization” is a sham; many CAM/IM therapies ultimately appeal to one of many examples of the “One True Cause of Disease” fallacy.
Speaking of shams, so is Dr. Weil’s claim of being “holistic”:
A third methodological issue that distinguishes IM from the environment of pharmaceutical RCTs involves systems thinking. With its roots in holistic, natural, and aboriginal medicine, IM has always embraced a more systems-based orientation to patient care than conventional care. It should come as no surprise that a narrow modality for evaluating treatment effectiveness would become increasingly limiting to IM research. The movement toward increasingly narrow scientific evaluations may create an artificial and arbitrary view of human health, medicine, and treatment effectiveness. Fritjof Capra, PhD, the well-known physicist, indicated that the great surprise of twentieth century science was that complex systems cannot be understood by analysis. Ecology and epigenetics are examples of the strong movement toward systems thinking in modern biology.
This is, of course, utter nonsense. “Holistic,” “natural,” and aboriginal medicine relied upon a prescientific, not a “holistic,” understanding of biology and patient care. Appealing to “systems” thinking based on prescientific or outright religious systems, such as homeopathic provings, traditional Chinese medicine, Ayurveda, and “energy healing,” is meaningless when the systems being invoked are without a grounding in science—or even reality. The rise of systems theory in biology and elsewhere has nothing to do with the sort of false “holistic” thinking invoked by Weil and his ilk. Rather, it has to do with the increasing ability to analyze more than one aspect of a complex system at once. For example, in my field (cancer), thirty years ago it was only possible to analyze one gene or perhaps handful of genes at a time. Then, beginning in the late 1990s, the development of cDNA microarray chips made it possible to analyze hundreds, then thousands of genes simultaneously—then every gene in the genome. As technology and computing power increased and scientists and mathematicians developed techniques to analyze ever larger datasets, it became possible to take the data from these sorts of experiments and begin to understand the complex networks inherent in the expression of the thousands of genes that are produced in human cells. Today, it is becoming increasingly possible to integrate massive quantities of data from various sources, including genomics, proteomics, and metabolomics and begin to understand the vastly complex networks that they form, how they resist perturbation, and how they can be restored when they are perturbed.
That is real holism. Not homeopathy. Not traditional Chinese medicine. Not Ayurveda. What Weil represents as “holism” is in reality a series of pretenders to “holistic” understanding that substitute non-evidence-based prescientific belief systems for science, gussying them up in “science-y”-sounding language that co-opts new science the way CAM/IM co-opts science-based modalities like diet and exercise as being somehow “alternative.”
The rest of the rest
Those who have actually read Dr. Weil’s article will note that I have not yet mentioned one point that Weil hammered on relentlessly, apparently thinking himself Maxwell bringing his silver hammer down upon the heads of those who think scientifically, the better to knock the propensity towards SBM (or even EBM) out of them. That issue is safety. Weil goes on and on about patient safety, even going so far as to propose a ranking system for patient safety. This part of his article is, quite frankly, an insult to physicians’ intelligence. Physicians practicing EBM already weigh efficacy/effectiveness versus safety; all Weil is doing is to rename and rebrand something that EBM does already. To a small extent he has a point that EBM does not yet have a as rigorous “hierarchy” of risk or harm to evaluate treatments as it does levels of evidence for efficacy, but even if we did have such a rigorous hierarchy, risk-benefit estimates still require weighing benefit versus risk. If the expected benefit is zero, or near zero, then even a tiny risk can quickly become unacceptable. To take one example, in the case of acupuncture, where the benefit is not distinguishable from that of placebo, then even a tiny risk of, say, a pneumothorax from a needle stuck too deep becomes very problematic. In other words, Weil’s classification system is unlikely to make CAM seem as attractive as he seems to think it will, although no doubt his intent is to make that evil “reductionistic” allopathic medicine seem more dangerous by comparison.
In addition, Weil makes a big show of complaining about commercial influence, in essence using a weaker form of the “pharma shill” gambit to dismiss pharma-funded RCTs whose conclusions he does not like. This serves, more than anything else, as a ploy to imply guilt by association, given that virtually all pharmaceuticals are manufactured for profit. As we have written about here on numerous occasions, commercial influence is a problem that is recognized in EBM (and SBM). It is increasingly appreciated as an issue, and practitioners of EBM do consider funding sources when considering clinical trial results. Indeed, one can’t help but wonder why Dr. Weil’s concern about pharma influence is so conveniently limited. Does he complain about the influence of supplement manufacturers (which, by the way, are increasingly owned by various pharmaceutical companies and wield a great deal of clout in Congress)? If he does, I haven’t seen evidence of it. Steve was right; this part is just window-dressing, as is Weil’s proposal for a different five-level hierarchy of evidence that not-so-subtly tilts the playing field so that some observational studies and other studies that can’t be considered RCTs could be considered Level One evidence.
What’s truly ironic is that Weil would never admit that there are few, if any, CAM modalities that can meet the criteria of level one evidence even under his own proposed system! In fact, most of the very best-supported CAM modalities would fall either into level four (“Weak indicators of uncertain value: poorly designed studies without strong support from the Hill criteria; small observational studies”) to level five (“Very weak indicators of efficacy or effectiveness: expert opinion of effectiveness; case series; multiple anecdotal reports”). That’s why it’s tempting simply to grant Weil his new system and then challenge him to show more than a pitiful handful of CAM modalities that have evidence to support them stronger than level four.
Finally, two of the more pernicious proposals in Weil’s paper are related. First, he repeats his explicit argument for favoring placebo medicine:
Given the history and philosophical preferences of allopathic medicine, it should come as no surprise that the factors defining the healing response become minimized or ignored in current practice. This failing must be remedied, as these factors account for a huge component of how humans heal and recover. The healing relationship has taken a much larger role in IM as practitioners in nearly all CAM modalities place a much higher emphasis on it. The importance of these issues can be demonstrated most clearly in psychiatry research, where the placebo response plays a huge role accounting for as much as 40%–90%+ of the total response.26
Significant placebo and expectancy responses inhabit other areas of medical practice, such as dealing with pain and even life expectancy in patients with terminal cancer. Clearly, patient factors must compose a significant part of all treatment selection processes. Ideally, every treatment should be matched to the individual’s belief system to reach the highest level of response possible.
Aside from stating, “CAM is placebo medicine, and we should use placebo medicine when it fits in with patient beliefs,” a more explicit admission that what is being proposed in this opinion piece is placebo medicine I cannot imagine. Worse, it’s based on misinformation, the most egregious of which is Weil’s claim that placebo and expectancy effects can increase life expectancy in patients with terminal cancer. Would that it were true, but unfortunately it’s not, either in late stage cancer or early stage cancer. Nor is there any evidence that it has an effect on recurrence. This is where Weil’s second, related thrust occurs. Using this assumption that placebo medicine “works” better if its tenets match the patient’s belief system, he proposes turning the entire concept of informed consent on its head with respect to medical decision-making (MDM):
The ideal process in MDM would remodel the process of informed consent. The actively engaged patient would be offered a quick overview of appropriate treatments (both CAM and conventional) with an unbiased reflection of both safety and effectiveness. The patient would then declare a preference for one over the other(s). Once a practitioner ascertains the basic worldview of a patient (natural versus conventional; safety versus effectiveness) many simple elements of MDM would flow quickly in the future.
This is a process that to which I’ve referred in the context of two other discussions, vaccines and supplements. In essence, what Dr. Weil is proposing is an ideology-based version of misinformed consent. Although Dr. Weil would not doubt strenuously object to my comparison, when you boil it all down, what he is doing, although more subtle, is not materially different than what anti-vaccine groups do when they grossly exaggerate the risks of vaccination and downplay its benefits or what “health freedom” groups do when they exaggerate the benefits (and the scientific evidence for such benefits) of supplements. What these groups each do is to give patients a skewed view of the risks and benefits of an intervention, intentionally making it more likely that patients will choose the “alternative” option. Weil is doing the same thing by trying to confuse the issue of efficacy/effectiveness by throwing in the issue of “patient belief” and placebo medicine. As is the case with anti-vaccine activists and supplement hawkers, the purpose is, having failed to win on “conventional science,” to find a way to tilt the playing field back towards the “alternative” using other means.
Conclusion
The proposals in Shannon et al are nothing original, although I must grudgingly tip my hat to Weil and his acolytes for having repackaged old ideas in a very attractive, slick new package. In essence, they boil down to a massive appeal to other ways of knowing, hence the attack on RCTs in which they point out deficiencies that those of us advocating SBM and EBM have recognized for decades; the emphasis on effectiveness over efficacy that ignores the fact that it’s necessary to have good evidence of efficacy before even considering effectiveness in the “real world”; the tarting up of safety considerations as though EBM doesn’t take safety into account; and a direct appeal to placebo medicine via the methodology of misinformed consent. All of these strategies are clearly designed to try to give CAM/IM a leg up on science that it can’t achieve through science alone. Unfortunately, although such strategies are transparent to SBM bloggers (and, I daresay, many of our readers), to others they are not so obvious, even to proponents of EBM. I also can’t help but think: If Dr. Weil detests EBM so much, he’s really not going to like SBM at all. Not one bit.
Perhaps the most ironic part of all of this is that many of us at SBM actually agree with the contention that EBM relies on RCTs too much. Many of us would even agree with several of the Bradford-Hill criteria as considerations that are worthy of more emphasis in EBM. Where we differ is in how to do this. Most importantly, we emphasize much more strongly scientific/biological plausibility. Prior probability appears to be anathema to Weil, given that he mentions it as part of the Bradford-Hill criteria and then completely ignores it thereafter. No doubt, he knows how scientifically implausible many CAM/IM modalities are. Instead, Dr. Weil Gish gallops off to emphasize, as he does in the title of his article, patient preferences and safety, in essence using these issues to bolster an explicit plea for placebo medicine. Rather than accepting that rigorous scientific examination of his favorite CAM/IM modalities fails to find any benefit over placebo, Dr. Weil cleverly embraces placebo medicine and argues for lower scientific standards to permit pseudoscientific and unscientific medicine to appear to be co-equal with EBM.
Unfortunately, such are the arguments that have been the wedge used to insert quackademic medicine into medical academia. Even more unfortunately, they are working.
REFERENCE:
Shannon, S., Weil, A., & Kaplan, B. (2011). Medical Decision Making in Integrative Medicine: Safety, Efficacy, and Patient Preference Alternative and Complementary Therapies, 17 (2), 84-91 DOI: 10.1089/act.2011.17210

   Attitudes and Public Health
Steven Novella
Link
Increasingly there is a cultural trend toward health care freedom and empowerment. This trend is partly a reaction to the paternalism of the past, and reflects an overall change in attitude by the public toward all institutions and authority. Within medicine there has also been a move toward the partnership model of practice – where patients are well-informed full partners in the decision-making process. But this trend has also been fueled by providers who want the public to have the freedom to choose their unconventional treatment, even if it does not meet reasonable standards for evidence or even ethics.
In addition the public must deal with an increasingly free health care market with an expanding array of products, and claims to back them up. The internet has served to facilitate and accelerate this process.
Therefore public education about common health matters is more important than ever. Part of the mission for this blog is to improve public health education, to correct common misconceptions, help put recent research into proper perspective, and to counter false or misleading propaganda or marketing claims. There seems to be an intense need for such correction, and mainstream media and the internet are full of misinformation. News outlets are a mixed-bag, sometimes providing helpful information, but more often emphasizing unusual or dramatic health risks while ignoring far more important but less interesting ones.
A new global survey reveals that the public is indeed largely misinformed about the risk factors for cancer, which is a reasonable marker for overall health care attitudes. Further, the survey shows that high-income countries are often more misinformed than low income countries. This might reflect the effect of marketing efforts made in higher-income countries.
Here are the key findings of the survey:
- People in high-income countries were the least likely to believe that drinking alcohol increases the risk of cancer. In that group, 42% said alcohol does not increase the risk. That compares with only 26% of respondents in middle-income countries and 15% in low-income countries saying that alcohol use does not increase the risk of cancer. In fact, cancer risk rises as alcohol intake increases.
- In high-income countries, the hazards of not eating enough fruits and vegetables scored more highly as a perceived risk (59%) than alcohol intake did (51%), even though the scientific evidence for the protective effect of fruit and vegetables is weaker than the evidence that alcohol intake is harmful.
- In rich countries, stress (57%) and air pollution (78%) scored higher as perceived risk factors for cancer than did alcohol intake. However, stress is not recognized as a cause of cancer and air pollution is a minor contributor compared with alcohol consumption.
- People in low- and middle-income countries have more pessimistic beliefs about cancer treatment than those in high-income countries. One of the more important problematic beliefs in lower-income countries concerned perceptions about the curability of cancer. The survey found that in such countries 48% said that “not much can be done” to cure cancer or that they didn’t know whether anything could be done. That compares with 39% in middle-income countries and 17% in high-income countries. Such a misbelief is worrying because it might deter people from participating in cancer screening programmes, which are important for saving lives.
- In general, people in all countries are more ready to accept that things outside of their control might cause cancer (such as air pollution), than things that are within their own control (such as overweight, which is a well-established cancer risk factor).
- An astonishing 75% percent of people in low-income countries said their preference was for their doctor to make all the treatment decisions. Only 8% said the doctor and patient should decide together and 9% said the patient should decide. That compares with a preference in rich countries for a more equitable decision-making style that emphasises self-determination, with 72% saying either that the decision should be made together or rest with the patient alone.
In countries like the US and the UK the public wants to be more involved in their own health care decisions and yet they were poorly informed about the scientific data regarding basic cancer risk factors.
It is not surprising to me that in wealthy countries people overestimate the impact of diet and stress on their cancer risk. This is in line with much “alternative” health marketing, which emphasizes all things “natural” as a marketing strategy. Improved nutrition and stress reduction are central to the claims of many dubious products and alternative ideologies.
There is also a great deal of propaganda aimed at sowing distrust in the institutions of medicine. Popular author Kevin Trudeau (who has been sanctioned by the FTC for false advertising), for example, has made millions telling the public not to trust their doctors, and that nutrition and “natural” treatments can prevent and even cure cancer. Scientific information about cancer risk factors is often diluted in a sea of misinformation, or is often dismissed as the self-serving lies of “the cancer treatment industry” or “big pharma” with the government as their lap dog.
This recent survey suggests that the propaganda is working and as a result the public has distorted and misleading beliefs about cancer risk factors – all in conjunction with the belief that people should be taking more personal responsibility for their health.
This points to the need for more efforts toward public education about basic health issues – especially those that affect their personal choices. How to deal with the misinformation is a much more difficult issue. As Kevin Trudeau discovered, he could easily get around the FTC sanctions by simply selling misinformation rather than products, because information is protected free-speech. There does not appear to be regulations in place to deal with the deliberate marketing of false or misleading information for its own sake (as the product itself) nor is it clear what form such regulations would or should take, if any.
For now the only remedy is to work toward a more skeptical and scientifically literate public.

   Science, Reason, Ethics, and Modern Medicine Part 4: Tu Quoque and History
Kimball Atwood
Link
Dr. Moran Weighs In
In last week’s post, I dubbed Dr. Peter Moran the “conscience” of SBM, citing his commitment to doing what’s best for individual patients even if, in theory at least, that may involve some manner of benign but fanciful treatments. I countered with my own opinion that honesty and integrity are necessary parts of any discussion with a patient, and that they, in turn, must not conflict with science and reason.* I added passages from a couple of key medical ethics treatises to support my assertion. Dr. Moran’s response, thoughtful and provocative as always, was buried in the midst of other commenters’ tangential arguments about the theory of evolution. Rather than continue its exile there, I reprint it here to give it the exposure that it deserves:
A blatant appeal to authority, but one that I mostly agree with. The difference between us is that I insist that medicine is about an infinite number of individual contexts and I see many examples where ethical absolutes (actually these are ethical guidelines rather than directives) do not apply or don’t seem to apply very well.
We scientists are ever-so cautious when making scientific judgments about complex matters; let’s not pretend that arriving at absolutes in medical ethics is a piece of cake, especially when it is not quite clear how anything done with the undiluted welfare of the individual patient in mind can be entirely unethical. I mean, why are we obliged to consider the impact of our decisions upon the fate of the planets (or whatever) when THIS patient needs help? In fact, at least one medical ethicist has gone so far as to state that it is not unethical for a doctor to prescribe a placebo treatment, so long as the doctor believes it will benefit the patient. I don’t quite agree with that bald statement — there should be a rider specifying that this may apply to *some* contexts where there is no obviously superior evidence-based method.
Here are some examples of the intellectual minefield we have to negotiate.
1. All the doctors I know would be prepared to call in the witch-doctor if it would help assuage the fears, or help in the management, of a seriously ill primitive tribesman. It seems we are prepared to pander to the superstitions of SOME cultures while despising any similar inclinations in our own.
2. I have previously asked this question which has to do with public policy in relation to safe “alternatives”. Take my word for it that every pharmacy in Europe displays “Homeopathie” (or language equivalents) in large letters outside. Would skeptics prefer those using such remedies for their minor and self-limiting complaints to be using NSAIDs or antibiotics or antidepressants instead, treatments that will often in such contexts themselves perform no better than placebo, but at substantially greater risks? Behind the usual healthfraud position there is both an exaggeration of the capacity of modern medicine and insufficient recognition of the harm that it can do. We definitely do not yet have entirely safe and 100% effective solutions to all of mankind’s ills, and certain imperfections of everyday medical practice can heighten the risks of the use of unnecessarily powerful pharmaceuticals. So what is the safest and most pragmatically realistic position here?
3. Following on from that — what is the evidence-based answer to non-specific tiredness and unhappiness? If people feel better for taking a multivitamin or an innocuous herb, why should we care? We keep on offering the public temporary answers to these things, prescribing (historically) amphetamines, cocaine, opiates, barbiturates and phenothiazines in massive quantities, only to take them away when problems such as addiction ensue. Is it right to then turn around and say, well you didn’t really need these things anyway, even denying them any relief that they may derive from “pretend medicines”. The science that matters will be argued out in other arenas.
That’s to give you some idea of the kind of thing that I am on about. You seem to think I am talking about doctors promoting CAM or placebo treatment as a matter of policy. I am not prepared to go that far, although I think I understand why some doctors might do that.
I agree that “medicine is about an infinite number of individual contexts and [there are] many examples where ethical absolutes do not apply or don’t seem to apply very well.” Nor did I really think that Dr. Moran was “talking about doctors promoting ‘CAM’ or placebo treatments as a matter of policy.” We disagree elsewhere, but he makes some interesting points.
The most compelling, in my opinion, is his point about “calling in witch doctors.” Yes, I would not be the first exception to that rule. I do have a response, however, and it has to do with the differences between some cultures and our own: the fallacy is that there are “similar inclinations.” There is a vast difference between the traditional, deeply held opinions of a primitive tribesman and his witch doctor, on the one hand, and the addled opinions of a born, raised, and (partially) educated American or European or Australian patient and his snake oil peddling quack on the other. Those in our culture, with occasional exceptions, were not brought up to believe in “CAM.” Many quacks are themselves physicians with no acceptable excuse, who until recently were kept in check by strong professional and governmental pressures, only to have found themselves reborn in a new “golden age of quackery.” Quacks who are not physicians have also been kept in check, until recently, by governmental pressures. Their newfound endorsement by governmental edict is no more justified than would be a similar endorsement of tribal witch doctors.
My major objection, as I’ve written in my bio and elsewhere on SBM, is to “implausible claims being promoted, tacitly or otherwise, by medical schools and government” and by physicians. Patients are not the problem; they are, if you’ll excuse the drama, the victims. Even seemingly benign practices, if implausible, can be harmful (“first, do no harm”), particularly when promoted by someone who should know better. This isn’t the setting for a complete discussion, but it will surely come on SBM. In the meantime, there is the unnecessary expense, a patient later feeling betrayed by her physician when the treatment doesn’t work or the “placebo effect” wears off and she realizes that it was only a “pretend medicine” all along, other patients or physicians concluding that the treatments have validity (and possibly being convinced to forgo rational treatment), people in general losing confidence in modern medicine when it appears that many physicians can’t distinguish between the rational and the bogus, wasted research funds, human subjects unnecessarily duped and endangered, a befuddled media, a befuddled public, befuddled leglislators endorsing quacks to a befuddled public, and more.
Yes, it is true that THIS patient needs help, but if physicians held exclusively to that ethic they wouldn’t have to concern themselves with antibiotic resistance, universal immunizations, judicious use of expensive but very low yield tests that are physically harmless and paid for with pooled money, or other cost-containment issues that don’t immediately affect THIS patient, would they? It’s true that there are many examples where ethical absolutes don’t seem to apply very well.
Moving on to number 2: every pharmacy in Europe that advertises “Homeopathie” is committing fraud, plain and simple. If physicians were more attentive to science and ethics and to explaining things to patients, politicians, and the media, more people would know that. The alternatives to homeopathy for minor or self-limiting complaints, moreover, need not be “NSAIDs or antibiotics or antidepressants” unless there is a good reason for one of them. The alternative is to explain to the patient that the complaint is minor or self-limiting! Why should honesty and integrity be so difficult? That some modern physicians prescribe potent drugs solely for placebo effects (and to get patients out of their offices) is an indictment of that practice, not an argument for “CAM.”
There are, moreover, plenty of rational, nonpharmacological treatments for all manner of minor or self-limiting complaints (and even for not so self-limiting complaints, such as chronic pain), e.g., gargling with hot water or sipping hot tea, time, avoiding environmental allergens, smoking cessation, weight loss, ice packs or heat packs, rest, exercise, massage (to make muscles feel better, not to “remove toxins”), pain management, physical therapy, eating more fibre, “sleep hygiene,” and many more.
The answer to point 3 is the same as the answer to point 2, and I certainly don’t mind if “people feel better for taking a multivitamin or an innocuous herb” (if that were the extent of “CAM” we’d hardly be wasting our time here). I just don’t think that physicians should make false claims. Point 3 also brings psychotherapy to mind. It need not be introduced in an unjustified or insulting way, i.e., by implying that the patient’s symptoms are “all in her head” or are “functional” or whatever. Rather, the honest physician will admit to not being able to explain the symptoms, but acknowledge that they must be distressing and suggest that it might be helpful to discuss how they impinge on the patient’s life and how she might better cope with them.
There are plenty of modern physicians whose patients are not disposed to look elsewhere, even if those patients have “nonspecific tiredness and unhappiness” and even if those physicians don’t suggest woo. Many patients, perhaps most, expect honesty and integrity, and have a pretty good sense of when they see it and when they don’t.
Dr. Moran also added another comment, reproduced in part here:
The ethical objection to ALL use of CAM, as presented by KA, is problematic because of the unspoken assumptions –
1. That conventional medicine has adequate answers to all health problems and the public should be content with what it offers. This applies fairly well to some illnesses but not many others.
2. That “alternative” methods do nothing worthwhile for those using them, anyway. This view is also context sensitive since considerable scientific evidence suggests the likelihood of symptom relief by the placebo medicines of CAM and from interactions with CAM practitioners.
3. That science provides a clear discrimination between what works and what doesn’t. Over the long haul that is true. But what do we mean by “works”? Where does the placebo complex of influences fit in?
Point number 1 is simply incorrect. I have not assumed, nor is the assumption inherent in ethical objections to physicians promoting ”CAM,” that “conventional medicine has adequate answers to all health problems.” I reject, as I hope I made clear above, the false dichotomy that for each health problem there is either a proven treatment or there is “CAM.”
Regarding points 2 and 3, I agree that the only “value” of almost all “CAM” treatments is in their “placebo effects.” But if a physician’s quest to elicit a placebo effect from an implausible treatment means pretending that the treatment has some other effect, specific to the patient’s problem, then that is a lie. So far, I’ve seen nothing to justify such a lie.
There is a potential placebo effect involved in every patient-practitioner interaction, and it needn’t require woo. The most important elements are that the practitioner be attentive and appear to give a damn. Many patients will experience dramatic improvements in symptoms just by finding out that whatever is causing them isn’t cancer or some other serious illness—which reflects an underappreciated reality about the goal of a diagnostic “workup”: not necessarily to determine what something is, but always to determine what it isn’t. The refusal of some physicians and patients to accept that truth is, in my opinion, an important basis for a “CAM” subset, “fad diagnoses” (perhaps a topic for another time).
As a patient, I have experienced two memorable examples of what I think of as “placebo effects.” The first occurred at the age of 18 when, plagued by adolescent angst, I visited a psychiatrist exactly once who told me, “you’re not crazy and you’re not going to go crazy.” I remained an adolescent for some time, but I sure felt a hell of a lot better. The second occurred many years later and was heralded not by a conversation, but by a diagnostic test: after a couple of months of excruciating neck pain, I had a magnetic resonance imaging (MRI) study that showed that my cervical spine was normal. As I read the report the pain abated, never to return to its previous degree of severity.
I am glad that Dr. Moran has described his views on this topic as “tentative positions that I am exploring.” Such candor serves to maintain my view of him as our “conscience,” even as it makes me optimistic that we might dissuade him from making those positions permanent. 
The Physician as Expert Consultant
Philosophers Clark Glymour and Douglas Stalker wrote an article 25 years ago for the New England Journal of Medicine titled “Engineers, cranks, physicians, magicians.” It was later reprinted as the first chapter of a book edited by the same authors, Examining Holistic Medicine, which deserves to be on the shelf of anyone interested in the topic. At the time the article was written, the terms “Complementary and Alternative Medicine” (“CAM”) and “Evidence-Based Medicine” (EBM) had yet to enter the lexicon. Other than that it is as timely now as it was then, and is also pertinent to much of the discussion here (note to Dr. Moran: this is no longer directed at you; I know that you are aware of these things). Because the article is not available online I’ve reproduced several passages:
Medicine in industrialized nations is scientific medicine. The claim tacitly made by American or European physicians, and tacitly relied on by their patients, is that their palliatives and procedures have been shown by science to be effective. Although the physician’s medical practice is not itself science, it is based on science and on training that is supposed to teach physicians to apply scientific knowledge to people in a rational way.
The practice of medicine in the United States and in other industrialized nations is a form of consultant engineering. The subjects are people rather than bridges, but in many respects the professions of medicine and engineering are alike. We expect skilled engineers to be able to learn from experience and to get better at building bridges, because we believe that their training has subjected them to a rational discipline that has made them good learners about such matters. Sometimes, of course, we are disappointed. It is entirely the same with physicians, who must apply both explicit scientific principles and also a great deal of tacit knowledge to the treatment of their patients. Medical training is supposed to make physicians good at applying scientific knowledge to sickness, and it is also supposed to make medical doctors good at acquiring through practice an abundance of tacit knowledge useful to their craft.
There is no reason, either historically or logically, to conceive of the science used by physician engineers as necessarily physical science. Engineers need not care in principle whether the generalizations on which they rely are psychological, physical, or psychophysical; what they care about is that the generalizations be applicable and that their reliability be scientifically demonstrated. A great deal of what physicians learn consists of biologic and biochemical generalizations, broadly construed, but they also learn a substantial body of psychophysical generalizations which can be regarded as bridging the crevasse between mind and body. For example, generalizations concerning the effects of drugs, correlating the location of pain with other physical symptoms of disease, and positing the causal factors in dizziness and senility connect the mental with the physical and are thus useful for medical engineering. If physicians learn relatively fewer generalizations that are entirely psychological or social in nature or that posit psychological mechanisms for physical effects, the reason is not that such generalizations are alien to the “medical model” but that relatively few of them are applicable and scientifically warranted.
There are alternative conceptions of the physician. Some of them play a dominant part in the understanding of medicine in other societies, and some serve to qualify the conception of the physician as engineer even in our own society. One such conception is that physicians are consolers. Another is that they are magicians who exercise occult powers to bring about healing. As magicians, they possess magical powers either because of the occult knowledge they possess or simply because of who they are—for example, because they stand in some special relation to gods or demons. Again, the physician may be understood to be someone who applies a reliable body of knowledge that is not warranted by science or by magic but is simply known and, so far as the community is concerned, always has been known. The warrant behind this conception of the healer is tradition and “common knowledge.”
These distinctions are more logical than sociological, and a society may combine several of them in the roles it assigns to healers. A medicine man can combine traditional therapies with magical claims, and both with a bit of consolation. A physician engineer can act as consoler; nothing in either logic or social psychology forbids it. But certain combinations are impossible or extraordinarily unlikely. A physician engineer cannot honestly claim powers of magic or occult knowledge. The principles governing scientific reasoning and belief are negative as well as positive, and they imply that occult doctrines are not worthy of belief. Moreover, physician engineers have no immunity to moral or ethical constraints. On the contrary, they are by training and by culture enmeshed in a tradition of rational thought about the obligations and responsibilities of their profession.
Is there another, holistic, conception of medicine distinct from those described above? Certainly, many people seem to think so. In 1978, a group of medical and osteopathic physicians formed the American Holistic Medicine Association, which now publishes a journal and whose meetings have been recognized for education purposes by the American Medical Association. Popular bookstores are filled with works on “holistic medicine,” many edited by medical doctors and some recommended by such political eminences as Edward Kennedy and George McGovern. The same shelves boast best-selling books on holistic medicine authored by professors at distinguished medal schools and, in at least one case, by a physician administrator at the National Institutes of Health. The therapies described and recommended in a typical book of the genre include biofeedback, hypnosis, psychic healing, chiropractic, tai chi, iridology, homeopathy, acupuncture, clairvoyant diagnosis, human auras, and Rolfing. One of the larger books of this kind was even subsidized by the National Institute of Mental Health.
What ties together the diverse practices…? In part, a banal rhetoric about the physician as consoler;… In part, familiar and rather useless admonitions about not overlooking the abundance of circumstances that may contribute to one condition or another. Such banalities are often true and no doubt sometimes ignored, with disastrous consequences, but they scarcely amount to a distinctive conception of medicine. Holist therapies can be divided into those that are adaptations of traditional medical practices in other societies—Chinese, Navajo, and so forth—and those that were invented, so to speak, the week before last by some relatively successful crank.
Insofar as it extends beyond banality, the holistic medical movement constitutes both a deliberate attempt to substitute a magical for an engineering conception of the physician and an attack on scientific understanding and reasoning. Although the holistic movement does not contain a conception of medicine distinct from those we have discussed, it does contain a reactionary impetus to return the practice of medicine to the practice of magic and to replace logic and method with occultism and obfuscation.
Several conceptions of “holism” have been developed in the writings of holistic practitioners and their advocates. Most of them are vacuous; they are banalities of orthodox medicine, or they have no medical content and no applicability to any possible practice of medicine; they merely sound nice. Some are patently false. A much-repeated and trivial thesis, and moreover one that is said to characterize the sense in which holistic medicine is “holistic,” amounts to no more than this: mental and physical properties are interdependent. Mental states affect physical states and physical states affect mental states. No one doubts it. To make such a claim seem somehow profound, holistic writers invariably conjoin it with a discussion of Cartesian dualism, insinuating that modern medicine follow Descartes in postulating an impassable chasm between the mind and the body. Modern medicine does no such thing, and could not even if it wanted to, since Descartes held no such view.
Another doctrine said to be holistic is that one’s state of health is affected by everything. Whatever this means, it has nothing to do with any possible practice of medicine, for no one can attend to everything. If physicians cannot distinguish relevant from irrelevant factors, important from unimportant causes, then they can do nothing. A variation of this doctrine is not vacuous but merely vapid: “Fundamental to holistic medicine is the recognition that each state of health and disease requires a consideration of all contributing factors: psychological, psychosocial, environmental, and spiritual.” [Pelletier 1979] This is not a new revelation about medicine. Insofar as such multiple factors are known and believed to be important, they are routinely addressed in conventional medicine practice. Patients who suffer from coronary heart disease may be treated with beta blockers and antiplatelet drugs, and they may also be advised to change their work, their diet, their smoking habits, their exercise habits, and their living conditions.
The essay discusses assertions that “ ‘all states of health and all disorders are considered to be psychosomatic’ [Pelletier 1977]“ and that “the entire body (and psyche) can be treated or diagnosed through the treatment or observations of a special part of the anatomy.” Because of such notions,
…holistic practitioners…are under no obligation, they believe, to reconcile their claims about therapy with what is known about the causal pathways of the body.
The holistic practitioner sees the body in much the same way that magicians of old viewed the universe. The body becomes the last bastion of magic.
The authors expose the fallacy of the argument that “holistic medicine has not really been investigated, that funds ought to be made available for conducting tests of holistic practices, and that meanwhile we should keep our minds open about holistic techniques.” Although they couldn’t have foreseen the scientific blind spot that would soon hamper academic medicine in the form of “evidence-based medicine,” the authors portend discussions of Prior Probability:
Although it might be interesting to know more about the physiological pathways that are correlated with such processes as the placebo effect, this has nothing do with taking seriously the claims advanced by iridologists or zone therapists or even chiropractors. The claim to diagnose by examining the eye or to cure by massaging the foot is completely bogus; we know more than enough about the workings of the body to be reasonably certain that geometric features of the iris, for instance, do not provide the specific information about disorders that iridologists claim they do….Of course, it is conceivable that the beliefs of scientific medicine are in error about one or another of these matters, but that is no reason for using public funds to investigate holistic claims. One cannot justify spending other people’s money simply because one can imagine something to be true. The mere fact that holistic medicine is widespread and enduring is no reason to take its claims seriously; superstition, self-deception, stupidity, and fraud are ubiquitous and always have been.
The authors refute the charge that “the lack of evidence concerning the specific curative powers of holistic therapies is the result of a conspiracy of disinterest.” They dispense with cultural relativism, another favorite refuge of IMC advocates, and demolish the ”differing paradigms of Thomas Kuhn” contrivance:
Holistic advocates repeatedly cite Kuhn and claim that holistic medicine is an alternative paradigm with its own standards, one that cannot be understood or assessed by the practitioners of orthodox medicine. If the claim were valid, holistic practice would have to constitute a scientific tradition, albeit one in competition with the tradition of orthodox medicine. However, holistic medicine is not a scientific tradition. It has no paradigmatic work, no recognized set of problems, and no shared standards for what constitutes a solution to those problems; it also lacks the critical exchange among its practitioners that is characteristic of the sciences. Cranks have been common throughout the history of science, as Kuhn, a distinguished historian of science, knew well. The work of cranks does not constitute a scientific revolution, and no cranks appear among Kuhn’s many examples.
Glymour and Stalker were prescient in their conclusion:
If holistic-health advocates were content with encouraging sensible preventive medicine or with criticizing the economic organization of American medicine, we might be enthusiastic, but they are not. If the movement were without influence on American life, we would be indifferent, but it is not. Holistic medicine is a pablum of common sense and nonsense offered by cranks and quacks and failed pedants who share an attachment to magic and an animosity toward reason. Too many people seem willing to swallow the rhetoric—even too many medical doctors—and the results will not be benign. At times, physicians may find themselves in sympathy with the holistic movement, because some fragment of the rhetoric rings true, because of certain practices and attitudes they encounter in their daily work with colleagues and patients, or because of dissatisfaction with the economic and social organization of medicine. One hopes they will speak bluntly, but it does no good to join forces with cranks and quacks, magicians and madmen.
Amen to that.
In last week’s post, I dubbed Dr. Peter Moran the “conscience” of SBM, citing his commitment to doing what’s best for individual patients even if, in theory at least, that may involve some manner of benign but fanciful treatments. I countered with my own opinion that honesty and integrity are necessary parts of any discussion with a patient, and that they, in turn, must not conflict with science and reason.* I added passages from a couple of key medical ethics treatises to support my assertion. Dr. Moran’s response, thoughtful and provocative as always, was buried in the midst of other commenters’ tangential arguments about the theory of evolution. Rather than continue its exile there, I reprint it here to give it the exposure that it deserves:
A blatant appeal to authority, but one that I mostly agree with. The difference between us is that I insist that medicine is about an infinite number of individual contexts and I see many examples where ethical absolutes (actually these are ethical guidelines rather than directives) do not apply or don’t seem to apply very well.
We scientists are ever-so cautious when making scientific judgments about complex matters; let’s not pretend that arriving at absolutes in medical ethics is a piece of cake, especially when it is not quite clear how anything done with the undiluted welfare of the individual patient in mind can be entirely unethical. I mean, why are we obliged to consider the impact of our decisions upon the fate of the planets (or whatever) when THIS patient needs help? In fact, at least one medical ethicist has gone so far as to state that it is not unethical for a doctor to prescribe a placebo treatment, so long as the doctor believes it will benefit the patient. I don’t quite agree with that bald statement — there should be a rider specifying that this may apply to *some* contexts where there is no obviously superior evidence-based method.
Here are some examples of the intellectual minefield we have to negotiate.
1. All the doctors I know would be prepared to call in the witch-doctor if it would help assuage the fears, or help in the management, of a seriously ill primitive tribesman. It seems we are prepared to pander to the superstitions of SOME cultures while despising any similar inclinations in our own.
2. I have previously asked this question which has to do with public policy in relation to safe “alternatives”. Take my word for it that every pharmacy in Europe displays “Homeopathie” (or language equivalents) in large letters outside. Would skeptics prefer those using such remedies for their minor and self-limiting complaints to be using NSAIDs or antibiotics or antidepressants instead, treatments that will often in such contexts themselves perform no better than placebo, but at substantially greater risks? Behind the usual healthfraud position there is both an exaggeration of the capacity of modern medicine and insufficient recognition of the harm that it can do. We definitely do not yet have entirely safe and 100% effective solutions to all of mankind’s ills, and certain imperfections of everyday medical practice can heighten the risks of the use of unnecessarily powerful pharmaceuticals. So what is the safest and most pragmatically realistic position here?
3. Following on from that — what is the evidence-based answer to non-specific tiredness and unhappiness? If people feel better for taking a multivitamin or an innocuous herb, why should we care? We keep on offering the public temporary answers to these things, prescribing (historically) amphetamines, cocaine, opiates, barbiturates and phenothiazines in massive quantities, only to take them away when problems such as addiction ensue. Is it right to then turn around and say, well you didn’t really need these things anyway, even denying them any relief that they may derive from “pretend medicines”. The science that matters will be argued out in other arenas.
That’s to give you some idea of the kind of thing that I am on about. You seem to think I am talking about doctors promoting CAM or placebo treatment as a matter of policy. I am not prepared to go that far, although I think I understand why some doctors might do that.
I agree that “medicine is about an infinite number of individual contexts and [there are] many examples where ethical absolutes do not apply or don’t seem to apply very well.” Nor did I really think that Dr. Moran was “talking about doctors promoting ‘CAM’ or placebo treatments as a matter of policy.” We disagree elsewhere, but he makes some interesting points.
The most compelling, in my opinion, is his point about “calling in witch doctors.” Yes, I would not be the first exception to that rule. I do have a response, however, and it has to do with the differences between some cultures and our own: the fallacy is that there are “similar inclinations.” There is a vast difference between the traditional, deeply held opinions of a primitive tribesman and his witch doctor, on the one hand, and the addled opinions of a born, raised, and (partially) educated American or European or Australian patient and his snake oil peddling quack on the other. Those in our culture, with occasional exceptions, were not brought up to believe in “CAM.” Many quacks are themselves physicians with no acceptable excuse, who until recently were kept in check by strong professional and governmental pressures, only to have found themselves reborn in a new “golden age of quackery.” Quacks who are not physicians have also been kept in check, until recently, by governmental pressures. Their newfound endorsement by governmental edict is no more justified than would be a similar endorsement of tribal witch doctors.
My major objection, as I’ve written in my bio and elsewhere on SBM, is to “implausible claims being promoted, tacitly or otherwise, by medical schools and government” and by physicians. Patients are not the problem; they are, if you’ll excuse the drama, the victims. Even seemingly benign practices, if implausible, can be harmful (“first, do no harm”), particularly when promoted by someone who should know better. This isn’t the setting for a complete discussion, but it will surely come on SBM. In the meantime, there is the unnecessary expense, a patient later feeling betrayed by her physician when the treatment doesn’t work or the “placebo effect” wears off and she realizes that it was only a “pretend medicine” all along, other patients or physicians concluding that the treatments have validity (and possibly being convinced to forgo rational treatment), people in general losing confidence in modern medicine when it appears that many physicians can’t distinguish between the rational and the bogus, wasted research funds, human subjects unnecessarily duped and endangered, a befuddled media, a befuddled public, befuddled leglislators endorsing quacks to a befuddled public, and more.
Yes, it is true that THIS patient needs help, but if physicians held exclusively to that ethic they wouldn’t have to concern themselves with antibiotic resistance, universal immunizations, judicious use of expensive but very low yield tests that are physically harmless and paid for with pooled money, or other cost-containment issues that don’t immediately affect THIS patient, would they? It’s true that there are many examples where ethical absolutes don’t seem to apply very well.
Moving on to number 2: every pharmacy in Europe that advertises “Homeopathie” is committing fraud, plain and simple. If physicians were more attentive to science and ethics and to explaining things to patients, politicians, and the media, more people would know that. The alternatives to homeopathy for minor or self-limiting complaints, moreover, need not be “NSAIDs or antibiotics or antidepressants” unless there is a good reason for one of them. The alternative is to explain to the patient that the complaint is minor or self-limiting! Why should honesty and integrity be so difficult? That some modern physicians prescribe potent drugs solely for placebo effects (and to get patients out of their offices) is an indictment of that practice, not an argument for “CAM.”
There are, moreover, plenty of rational, nonpharmacological treatments for all manner of minor or self-limiting complaints (and even for not so self-limiting complaints, such as chronic pain), e.g., gargling with hot water or sipping hot tea, time, avoiding environmental allergens, smoking cessation, weight loss, ice packs or heat packs, rest, exercise, massage (to make muscles feel better, not to “remove toxins”), pain management, physical therapy, eating more fibre, “sleep hygiene,” and many more.
The answer to point 3 is the same as the answer to point 2, and I certainly don’t mind if “people feel better for taking a multivitamin or an innocuous herb” (if that were the extent of “CAM” we’d hardly be wasting our time here). I just don’t think that physicians should make false claims. Point 3 also brings psychotherapy to mind. It need not be introduced in an unjustified or insulting way, i.e., by implying that the patient’s symptoms are “all in her head” or are “functional” or whatever. Rather, the honest physician will admit to not being able to explain the symptoms, but acknowledge that they must be distressing and suggest that it might be helpful to discuss how they impinge on the patient’s life and how she might better cope with them.
There are plenty of modern physicians whose patients are not disposed to look elsewhere, even if those patients have “nonspecific tiredness and unhappiness” and even if those physicians don’t suggest woo. Many patients, perhaps most, expect honesty and integrity, and have a pretty good sense of when they see it and when they don’t.
Dr. Moran also added another comment, reproduced in part here:
The ethical objection to ALL use of CAM, as presented by KA, is problematic because of the unspoken assumptions –
1. That conventional medicine has adequate answers to all health problems and the public should be content with what it offers. This applies fairly well to some illnesses but not many others.
2. That “alternative” methods do nothing worthwhile for those using them, anyway. This view is also context sensitive since considerable scientific evidence suggests the likelihood of symptom relief by the placebo medicines of CAM and from interactions with CAM practitioners.
3. That science provides a clear discrimination between what works and what doesn’t. Over the long haul that is true. But what do we mean by “works”? Where does the placebo complex of influences fit in?
Point number 1 is simply incorrect. I have not assumed, nor is the assumption inherent in ethical objections to physicians promoting ”CAM,” that “conventional medicine has adequate answers to all health problems.” I reject, as I hope I made clear above, the false dichotomy that for each health problem there is either a proven treatment or there is “CAM.”
Regarding points 2 and 3, I agree that the only “value” of almost all “CAM” treatments is in their “placebo effects.” But if a physician’s quest to elicit a placebo effect from an implausible treatment means pretending that the treatment has some other effect, specific to the patient’s problem, then that is a lie. So far, I’ve seen nothing to justify such a lie.
There is a potential placebo effect involved in every patient-practitioner interaction, and it needn’t require woo. The most important elements are that the practitioner be attentive and appear to give a damn. Many patients will experience dramatic improvements in symptoms just by finding out that whatever is causing them isn’t cancer or some other serious illness—which reflects an underappreciated reality about the goal of a diagnostic “workup”: not necessarily to determine what something is, but always to determine what it isn’t. The refusal of some physicians and patients to accept that truth is, in my opinion, an important basis for a “CAM” subset, “fad diagnoses” (perhaps a topic for another time).
As a patient, I have experienced two memorable examples of what I think of as “placebo effects.” The first occurred at the age of 18 when, plagued by adolescent angst, I visited a psychiatrist exactly once who told me, “you’re not crazy and you’re not going to go crazy.” I remained an adolescent for some time, but I sure felt a hell of a lot better. The second occurred many years later and was heralded not by a conversation, but by a diagnostic test: after a couple of months of excruciating neck pain, I had a magnetic resonance imaging (MRI) study that showed that my cervical spine was normal. As I read the report the pain abated, never to return to its previous degree of severity.
I am glad that Dr. Moran has described his views on this topic as “tentative positions that I am exploring.” Such candor serves to maintain my view of him as our “conscience,” even as it makes me optimistic that we might dissuade him from making those positions permanent. 
The Physician as Expert Consultant
Philosophers Clark Glymour and Douglas Stalker wrote an article 25 years ago for the New England Journal of Medicine titled “Engineers, cranks, physicians, magicians.” It was later reprinted as the first chapter of a book edited by the same authors, Examining Holistic Medicine, which deserves to be on the shelf of anyone interested in the topic. At the time the article was written, the terms “Complementary and Alternative Medicine” (“CAM”) and “Evidence-Based Medicine” (EBM) had yet to enter the lexicon. Other than that it is as timely now as it was then, and is also pertinent to much of the discussion here (note to Dr. Moran: this is no longer directed at you; I know that you are aware of these things). Because the article is not available online I’ve reproduced several passages:
Medicine in industrialized nations is scientific medicine. The claim tacitly made by American or European physicians, and tacitly relied on by their patients, is that their palliatives and procedures have been shown by science to be effective. Although the physician’s medical practice is not itself science, it is based on science and on training that is supposed to teach physicians to apply scientific knowledge to people in a rational way.
The practice of medicine in the United States and in other industrialized nations is a form of consultant engineering. The subjects are people rather than bridges, but in many respects the professions of medicine and engineering are alike. We expect skilled engineers to be able to learn from experience and to get better at building bridges, because we believe that their training has subjected them to a rational discipline that has made them good learners about such matters. Sometimes, of course, we are disappointed. It is entirely the same with physicians, who must apply both explicit scientific principles and also a great deal of tacit knowledge to the treatment of their patients. Medical training is supposed to make physicians good at applying scientific knowledge to sickness, and it is also supposed to make medical doctors good at acquiring through practice an abundance of tacit knowledge useful to their craft.
There is no reason, either historically or logically, to conceive of the science used by physician engineers as necessarily physical science. Engineers need not care in principle whether the generalizations on which they rely are psychological, physical, or psychophysical; what they care about is that the generalizations be applicable and that their reliability be scientifically demonstrated. A great deal of what physicians learn consists of biologic and biochemical generalizations, broadly construed, but they also learn a substantial body of psychophysical generalizations which can be regarded as bridging the crevasse between mind and body. For example, generalizations concerning the effects of drugs, correlating the location of pain with other physical symptoms of disease, and positing the causal factors in dizziness and senility connect the mental with the physical and are thus useful for medical engineering. If physicians learn relatively fewer generalizations that are entirely psychological or social in nature or that posit psychological mechanisms for physical effects, the reason is not that such generalizations are alien to the “medical model” but that relatively few of them are applicable and scientifically warranted.
There are alternative conceptions of the physician. Some of them play a dominant part in the understanding of medicine in other societies, and some serve to qualify the conception of the physician as engineer even in our own society. One such conception is that physicians are consolers. Another is that they are magicians who exercise occult powers to bring about healing. As magicians, they possess magical powers either because of the occult knowledge they possess or simply because of who they are—for example, because they stand in some special relation to gods or demons. Again, the physician may be understood to be someone who applies a reliable body of knowledge that is not warranted by science or by magic but is simply known and, so far as the community is concerned, always has been known. The warrant behind this conception of the healer is tradition and “common knowledge.”
These distinctions are more logical than sociological, and a society may combine several of them in the roles it assigns to healers. A medicine man can combine traditional therapies with magical claims, and both with a bit of consolation. A physician engineer can act as consoler; nothing in either logic or social psychology forbids it. But certain combinations are impossible or extraordinarily unlikely. A physician engineer cannot honestly claim powers of magic or occult knowledge. The principles governing scientific reasoning and belief are negative as well as positive, and they imply that occult doctrines are not worthy of belief. Moreover, physician engineers have no immunity to moral or ethical constraints. On the contrary, they are by training and by culture enmeshed in a tradition of rational thought about the obligations and responsibilities of their profession.
Is there another, holistic, conception of medicine distinct from those described above? Certainly, many people seem to think so. In 1978, a group of medical and osteopathic physicians formed the American Holistic Medicine Association, which now publishes a journal and whose meetings have been recognized for education purposes by the American Medical Association. Popular bookstores are filled with works on “holistic medicine,” many edited by medical doctors and some recommended by such political eminences as Edward Kennedy and George McGovern. The same shelves boast best-selling books on holistic medicine authored by professors at distinguished medal schools and, in at least one case, by a physician administrator at the National Institutes of Health. The therapies described and recommended in a typical book of the genre include biofeedback, hypnosis, psychic healing, chiropractic, tai chi, iridology, homeopathy, acupuncture, clairvoyant diagnosis, human auras, and Rolfing. One of the larger books of this kind was even subsidized by the National Institute of Mental Health.
What ties together the diverse practices…? In part, a banal rhetoric about the physician as consoler;… In part, familiar and rather useless admonitions about not overlooking the abundance of circumstances that may contribute to one condition or another. Such banalities are often true and no doubt sometimes ignored, with disastrous consequences, but they scarcely amount to a distinctive conception of medicine. Holist therapies can be divided into those that are adaptations of traditional medical practices in other societies—Chinese, Navajo, and so forth—and those that were invented, so to speak, the week before last by some relatively successful crank.
Insofar as it extends beyond banality, the holistic medical movement constitutes both a deliberate attempt to substitute a magical for an engineering conception of the physician and an attack on scientific understanding and reasoning. Although the holistic movement does not contain a conception of medicine distinct from those we have discussed, it does contain a reactionary impetus to return the practice of medicine to the practice of magic and to replace logic and method with occultism and obfuscation.
Several conceptions of “holism” have been developed in the writings of holistic practitioners and their advocates. Most of them are vacuous; they are banalities of orthodox medicine, or they have no medical content and no applicability to any possible practice of medicine; they merely sound nice. Some are patently false. A much-repeated and trivial thesis, and moreover one that is said to characterize the sense in which holistic medicine is “holistic,” amounts to no more than this: mental and physical properties are interdependent. Mental states affect physical states and physical states affect mental states. No one doubts it. To make such a claim seem somehow profound, holistic writers invariably conjoin it with a discussion of Cartesian dualism, insinuating that modern medicine follow Descartes in postulating an impassable chasm between the mind and the body. Modern medicine does no such thing, and could not even if it wanted to, since Descartes held no such view.
Another doctrine said to be holistic is that one’s state of health is affected by everything. Whatever this means, it has nothing to do with any possible practice of medicine, for no one can attend to everything. If physicians cannot distinguish relevant from irrelevant factors, important from unimportant causes, then they can do nothing. A variation of this doctrine is not vacuous but merely vapid: “Fundamental to holistic medicine is the recognition that each state of health and disease requires a consideration of all contributing factors: psychological, psychosocial, environmental, and spiritual.” [Pelletier 1979] This is not a new revelation about medicine. Insofar as such multiple factors are known and believed to be important, they are routinely addressed in conventional medicine practice. Patients who suffer from coronary heart disease may be treated with beta blockers and antiplatelet drugs, and they may also be advised to change their work, their diet, their smoking habits, their exercise habits, and their living conditions.
The essay discusses assertions that “ ‘all states of health and all disorders are considered to be psychosomatic’ [Pelletier 1977]“ and that “the entire body (and psyche) can be treated or diagnosed through the treatment or observations of a special part of the anatomy.” Because of such notions,
…holistic practitioners…are under no obligation, they believe, to reconcile their claims about therapy with what is known about the causal pathways of the body.
The holistic practitioner sees the body in much the same way that magicians of old viewed the universe. The body becomes the last bastion of magic.
The authors expose the fallacy of the argument that “holistic medicine has not really been investigated, that funds ought to be made available for conducting tests of holistic practices, and that meanwhile we should keep our minds open about holistic techniques.” Although they couldn’t have foreseen the scientific blind spot that would soon hamper academic medicine in the form of “evidence-based medicine,” the authors portend discussions of Prior Probability:
Although it might be interesting to know more about the physiological pathways that are correlated with such processes as the placebo effect, this has nothing do with taking seriously the claims advanced by iridologists or zone therapists or even chiropractors. The claim to diagnose by examining the eye or to cure by massaging the foot is completely bogus; we know more than enough about the workings of the body to be reasonably certain that geometric features of the iris, for instance, do not provide the specific information about disorders that iridologists claim they do….Of course, it is conceivable that the beliefs of scientific medicine are in error about one or another of these matters, but that is no reason for using public funds to investigate holistic claims. One cannot justify spending other people’s money simply because one can imagine something to be true. The mere fact that holistic medicine is widespread and enduring is no reason to take its claims seriously; superstition, self-deception, stupidity, and fraud are ubiquitous and always have been.
The authors refute the charge that “the lack of evidence concerning the specific curative powers of holistic therapies is the result of a conspiracy of disinterest.” They dispense with cultural relativism, another favorite refuge of IMC advocates, and demolish the ”differing paradigms of Thomas Kuhn” contrivance:
Holistic advocates repeatedly cite Kuhn and claim that holistic medicine is an alternative paradigm with its own standards, one that cannot be understood or assessed by the practitioners of orthodox medicine. If the claim were valid, holistic practice would have to constitute a scientific tradition, albeit one in competition with the tradition of orthodox medicine. However, holistic medicine is not a scientific tradition. It has no paradigmatic work, no recognized set of problems, and no shared standards for what constitutes a solution to those problems; it also lacks the critical exchange among its practitioners that is characteristic of the sciences. Cranks have been common throughout the history of science, as Kuhn, a distinguished historian of science, knew well. The work of cranks does not constitute a scientific revolution, and no cranks appear among Kuhn’s many examples.
Glymour and Stalker were prescient in their conclusion:
If holistic-health advocates were content with encouraging sensible preventive medicine or with criticizing the economic organization of American medicine, we might be enthusiastic, but they are not. If the movement were without influence on American life, we would be indifferent, but it is not. Holistic medicine is a pablum of common sense and nonsense offered by cranks and quacks and failed pedants who share an attachment to magic and an animosity toward reason. Too many people seem willing to swallow the rhetoric—even too many medical doctors—and the results will not be benign. At times, physicians may find themselves in sympathy with the holistic movement, because some fragment of the rhetoric rings true, because of certain practices and attitudes they encounter in their daily work with colleagues and patients, or because of dissatisfaction with the economic and social organization of medicine. One hopes they will speak bluntly, but it does no good to join forces with cranks and quacks, magicians and madmen.
Amen to that.

   The “pharma shill” gambit
David Gorski
Link
I’ve mentioned before on this blog at least once that I cut my skeptical teeth, so to speak, on Usenet, that vast untamed and largely unmoderated territory full of tens of thousands of discussion newsgroups which used to be a lot more active before the rise of the World Wide Web and then later blogs. These days, few ISPs even offer much in the way of Usenet access; it’s become pretty much irrelevant since Google archived Usenet in the form of Google Groups. My forays into skepticism started out with combatting Holocaust denial on a newsgroup known as alt.revisionism (as good an excuse as any to remind you that nearly all Holocaust “revisionism” isn’t historical revisionism but is actually denial) and then branched out into more general skepticism, particularly about the claims of creationists and, of course, promoters of “alternative” medicine, the latter of which ultimately led me to being the editor of this wild and woolly thing we call the Science-Based Medicine blog. After I began to participate in the debates in the main newsgroup where alternative medicine is discussed, misc.health.alternative, it didn’t take me long to encounter a favorite tactic of promoters of alt-med who were not happy with one who insists on evidence-based medicine and who therefore questions claims that are obviously not based in valid science: The “Pharma Shill” Gambit. This is a technique of ad hominem attack in which a defender of “alternative” medicine, offended by your questioning of, for instance, his/her favorite herb, colon or liver flush technique, zapper, or cancer “cure,” tries to “poison the well” by implying or outright stating you must be in the pay of a pharmaceutical company, hired for nefarious purposes.
Since I entered the blogosphere several years ago under another guise, I’ve only occasionally checked back at my old stomping ground, mainly because blogging is so much less constraining than posting to Usenet, where mostly I used to respond to the posts of others, rather than writing about what I wanted to write about. A while back, though, out of curiosity I checked back and found this interesting little tidbit from a poster calling himself PeterB that demonstrated such a perfect example of the “pharma shill” gambit that I thought it might serve as a perfect example of the sort of thing I’ve had to put up with ever since I started speaking out against quackery:
To : All participants and readers of misc.health.alternative + other health-related newsgroups
Please be aware that many comments and responses posted to this forum are not those of casual posters interested in an honest exchange. A number of individuals with ties to industry are engaging an effort to shape public sentiment about the risks of mainstream medicine while denigrating the benefits and validity of natural medicine. I refer to these individuals broadly as “Pharma Bloggers”(*). Pharma Bloggers on usenet don’t promote a specific company or product, as might be the case with standard “blogging” on a weblog. Most of these people are likely to have an association with a PR campaign whose “blogging” efforts are underwritten by the media and marketing groups of industry. They are not difficult to identify due to specific patterns of behaviour in posting.
Here are a few points to remember while participating in usenet newsgroups:
     1.  Pharma Bloggers on usenet use intimidation, mockery, and insults to silence those who express belief or interest in natural medicine.
     2.  Pharma Bloggers on usenet attack those who question the effectiveness of mainstream medicine and defend disease-management “healthcare” as the only viable form of medicine.
     3.  Pharma Bloggers on usenet post the majority of their responses simply to bury the comments of others; they also strive obsessively to have the last word.
     4.  Pharma Bloggers on usenet are much faster at posting than casual participants; they almost always respond first to a new thread, question, or observation.
     5.  Pharma Bloggers on usenet use multiple “bloggers” in a swap-&-relay fashion to create an aura of the “consensus view” in an effort to isolate posters who question the value of mainstream medicine. You will see this tactic used more often than any other.
Tip: If you find yourself reading a response that is unusually dramatic in tone, or inexplicably vicious toward other posters, and if that response is a defense of mainstream medicine, you can be sure you have stumbled upon a “Pharma Blogger.” Unfortunately, there are more of these individuals posting to usenet on a daily basis than virtually anyone else, which is why I am posting this alert. If you find it odd that so few people on health-related usenet newsgroups are expressing an interest in natural medicine, it isn’t because they aren’t there, it’s because they have been intimidated into silence. The Pharma Bloggers have over-run the various newsgroups with their industrial brand of dogma, mockery, and ridicule. Many casual posters are simply frightened away. That’s one of the goals of Pharma Blogging.
(*) Pharma Blogger: An individual who uses the Internet to: 1) promote and defend maintstream medicine while denigrating natural medicine approaches; 2) attack others who express a preference for natural medicine, or who question the value of mainstream medicine; and 3) cite a variety of “junk medical science” funded by industry for the purpose of establishing markets for marginally effective, and often dangerous, medical products and devices.
PeterB
See what I had to deal with? First, let me just mention that I realize that astroturf campaigns do exist, but, quite frankly, die-hard defenders of alt-med on Usenet like PeterB tend to be interested in such Internet PR efforts only as a means of smearing those who criticize them for their claims or who have the temerity to ask them to provide scientific studies to back up their assertions. To them, everyone who questions them is probably part of an astroturf campaign. It goes with the conspiracy-mongering proclivities so common among cranks.
This sort of obvious pre-emptive ad hominem attack (again, a.k.a. the logical fallacy of poisoning the well) would be utterly laughable if it were not so common. I sometimes get the impression that PeterB and his compatriots must think that there are hordes of “pharma shills” sitting behind banks of computers (remember the claim “more of these individuals posting to Usenet than anyone else”), waiting to pounce the instant anyone like PeterB starts posting critiques of big pharma or praising herbal “cures.” (Yes, that they seem to think they are worth that sort of effort implies PeterB and others like them do seem to have an inflated view of their own importance.) My usual first response to such gambits tends to be facetious and runs along the lines of asking, “Where do I sign up to become a pharma shill? How do I get me a piece of that action? After all, why should I waste my time seeing patients and working like a dog to do science, publish papers, and write grants and then only having a couple of hours in the evenings to blog, when I could make big bucks ruthlessly mocking online dissent against big pharma full time while sitting back in my pajamas and sipping a big hot mug of coffee? Count me in!” (Expect to see my words posted somewhere out of context to make it seem as though I was being serious about this.)
However, facetiousness usually just infuriates people like PeterB to new heights of “pharma shill” accusations. At that point, it’s time to try to be rational, hard as it may be in the face of such provocation, but I try. First, a lot of this smear tends to be a case of projection, of the pot calling the kettle black. For example, #1, #2, and #3 are more typical of supporters of alt-med than of anyone who questions alt-med claims. Indeed, the denizens of misc.health.alternative who are most pro-alternative medicine tend to react quite defensively to questioning of their assertions. They are often like a group of Cyber Sisters (except that they are comprised of both men and women) ruthlessly descending upon anyone who questions the dogma of their favorite alternative medicine, criticizes their behavior, or suggests that maybe, just maybe, conventional medicine might have value. (No, those on “our side” are not entirely innocent, but in my experience the certain promoters of alt-med tend to be quicker with the ad hominem.) One reason for this, I suspect, is that many of them are also active on moderated groups such as CureZone.com, where anyone questioning the alt-med treatment du jour too long or too vigorously will be banned from the discussion groups, thus providing a nice, safe, cuddly environment, where never is heard a discouraging word towards quackery. #4 and #5 are clearly designed to imply that the so-called “Pharma Bloggers” either don’t have a regular job (why else would they have so much time?) or that they are working for big pharma. Of course, they never provide any evidence to support their accusations. In fact, they almost never provide even any reasoning to support their accusations more substantive than variations on “he’s criticizing alternative medicine a lot so he must be a pharma shill.”
The “pharma shill” gambit, like other varieties of ad hominem or well-poisoning rhetoric, conveniently frees defenders of “alternative” medicine from having to argue for their favorite remedies on the science and clinical studies supporting them (which in most cases tend to be badly designed or nonexistent). It’s a technique that’s not just limited to them, either. Anti-vaccination cranks and mercury/autism conspiracy theorists like it too, and, indeed, I have been the subject of some particularly vicious attacks over the years at the hands of the anti-vaccine movement1,2, Generation Rescue and its founder J. B. Handley in particular3,4,5, all of whom appear to be trying very hard to poison my Google reputation. (Indeed, another such attack rolled in just today, courtesy of AoA hanger-on Harold Doherty.) Moreover, J.B. Handley, in particular, has also attacked our very own Steve Novella. More recently, the late Hulda Clark’s former attack Chihuahua Tim Bolen appears to have decided for some reason that I have been named the heir apparent to Dr. Stephen Barrett. While I’m flattered that he somehow seems to think this, I’ll have to be at this for many more years and become much more effective even to approach Barrett’s legacy. Be that as it may, you have to have a tough skin if you’re going to try to combat the infiltration of pseudoscience into medicine.
Skeptico pointed out, even if a newsgroup denizen were a pharma shill, that wouldn’t necessarily invalidate his argument. Yes, in the case of a true “shill” who does not reveal that he works for a pharmaceutical company and pretends to be “objective,” it would be entirely appropriate to “out” that person with extreme prejudice, so that his bias could be taken into account. That being said, I’ll take this opportunity to point out that I have never over the last decade observed such a person in action, which tells me that they are probably a lot less common than people like PeterB like to claim. Even in the case of a real shill, however, this sort of “outing” is not a refutation of that person’s arguments; it merely serves to increase appropriately the level of skepticism about what that person is saying. Such an “outing” still leaves the task of actually using evidence, logic, and sound arguments to refute what that person is saying, something boosters of alt-med rarely even attempt to do. It’s far easier to fling the accusation of “pharma shill” about and see if they can get it to stick, as PeterB and his ilk do.
No doubt, may of our readers here at SBM have heard of Godwin’s Law, which states:
As an online discussion grows longer, the probability of a comparison involving Nazis or Hitler approaches 1.
Over the years, an assumed corollary to Godwin’s Law has become commonly accepted that in essence states that once such an “argumentum ad Nazium” is made, the discussion thread is over, and whoever brought up the Hitler or Nazi analogy first automatically loses the debate. Recently, this corollary was the basis of Scopie’s Law, which is more relevant to the topics covered on this blog and states:
In any discussion involving science or medicine, citing Whale.to as a credible source loses you the argument immediately …and gets you laughed out of the room.
Over time, I and others have suggested that, although Whale.to is clearly The One Quack Site To Rule Them All, there are at least a few other sites whose promotion of pseudoscience lead them to deserve to be included in Scopie’s Law, such as NaturalNews.com, Mercola.com, and the anti-vaccine crank blog Age of Autism. However, I think that there’s another law, a law similar to the original formulation of Godwin’s Law, that applies to virtually any online discussion of anything resembling alternative medicine. Perhaps we could dub it “Gorski’s Law,” and this is what I propose it to say:
As an online discussion of health, in particular vaccines or alternative medicine, grows longer, the probability of the invocation of the ‘pharma shill gambit’ approaches one.
If there’s an exception to this law, other than in moderated forums and (usually but not always) here at SBM, I haven’t found it yet.

   In which Dr. Gorski once again finds himself a target of the “pharma shill” gambit
David Gorski
Link
The “pharma shill gambit”: The quack’s favorite flavor of ad hominem argument
One of the very favorite and most commonly used tactics to attack criticism in the armamentarium of pseudoscientists, cranks, and quacks (not to mention politicians) is the ad hominem fallacy. In this fallacy, rather than addressing the actual evidence and science that demonstrate their favorite brand of woo to be nothing more than fairy dust, the idea is to preemptively attack and discredit the person. The ad hominem is not just insults or concluding that someone is ignorant because, well, they say ignorant things and make stupid arguments (in which case calling someone stupid or ignorant might just be drawing a valid, albeit impolitic, conclusion from observations of that person’s behavior), but rather arguing or insinuating that you shouldn’t accept someone’s arguments not because their arguments are weak but because they have this personal characteristic or that or belong to this group or that. Truly, the ad hominem is right up there with demanding public “debates” with skeptics as a favored defense strategy of cranks of all stripes.
Among the very favorite flavors of ad hominem attack used by quacks, cranks, and pseudoscientists is the fallacy of poisoning the well. This particular fallacy alludes to the medieval European myth that the Black Plague was caused by Jews poisoning town wells. Not surprisingly, this myth was used as a justification for pogroms and the persecution of the Jews. The idea is to poison how others view your opponent by preemptively attacking them. Well do I know this fallacy, having been at the receiving end of it many times! Basically, it involves invoking something bad or biased about a person’s situation or personality and then using a phrase something like, “Of course he (or she) would say that” to dismiss a person’s arguments, the implication being that the person receives such benefits from holding the position being attacked or has such a personality that he couldn’t argue otherwise regardless of the evidence. In my admittedly anecdotal experience, far and away the most common use of the ad hominem from quacks and pseudoscientists is what I once described as “the pharma shill gambit.” The idea behind this gambit when it comes to attacking those of us who promote science-based medicine is to tar one’s opponent as being a “shill” for big pharma or claiming that we have a conflict of interest so blatant that “of course we would say that.” In most cases, the bogey man is big pharma, in whose pockets we SBM bloggers are supposed to be safely (and profitably) ensconced, blogging away in our underwear for big bucks and, following the orders of our supposed paymasters, attacking anything that has even a whiff of being “alternative” or that “questions” the safety and/or efficacy of vaccines.
While I realize that there is such a thing as an “astroturf” campaign, in the vast majority of cases, the pharma shill gambit is nothing more than the variant of the ad hominem fallacy known as poisoning the well. I also realize that conflicts of interest (COIs) matter, particularly undisclosed COIs. Indeed, I wrote a rather lengthy post (I know, I know, do I write any other length of post?) about 8 months ago laying out my views regarding COIs in science-based medicine. The short version is that we all have COIs of some sort or another, be they financial, belief-based, or emotional, and more disclosure is usually better, to let the reader decide for himself. As far as COIs related to big pharma or finances, I think Mark Crislip put it quite well in his most recent Quackcast when he said that if a study is funded by big pharma, he decreases the strength of the evidence in his mind by a set amount. However, evidence is evidence, and, although it is reasonable to increase one’s level of skepticism if there is a major COI involving the authors, be it big pharma or otherwise, it is not reasonable to use that COI as the sole reason for rejecting its findings out of hand. That’s just an intellectually lazy excuse to dismiss the study, nothing more. Indeed, one prominent difference between a scientist and a pseudoscientist or quack is that in general scientists understand this and struggle to assign the correct degree of skepticism due to a COI when analyzing scientific studies, while quacks and pseudoscientists do not. It’s far easier for them just to put their fingers in their ears and scream “Conflict of interest! Conflict of interest!” and then use that to dismiss completely their opponent’s argument. It’s simple, neat, and it doesn’t require all that nasty thinking and weighing of evidence..
Yes, quacks, cranks, and pseudoscientists really, really like their “poisoning the well”-style ad hominem attacks. So much do they rely on such attacks that, when they can’t find a real COI to use, abuse, and exaggerate, they’ll make one up, hence the pharma shill gambit. Certainly they’ve tried it on me many times. Over the last six years I’ve lost track of how many times anti-vaccine zealots, homeopaths, Suzanne Somers and Jenny McCarthy fans, HIV/AIDS denialists, and supporters of “alternative medicine” have asked me accusingly, “Who do you work for?” or outright accused me, “I bet you work for Merck” or [insert most hated pharmaceutical company here — I used Merck because of the whole Vioxx issue]. Then they don’t believe me when I point out to them that, sadly, the amount of funding I receive from any medical or pharmaceutical company is…..[drum roll, please]….zero!
That’s right. Zero. Nada. Zip. Sadly, being a high-ranking mouthpiece of big pharma doesn’t pay what people think it pays. Clearly, I need a new agent. Or just an agent. And a ruthless editor, come to think of it. Be that as it may, over the weekend I became aware that apparently I’m once again in the sights of the anti-vaccine propaganda blog Age of Autism and that the apparent weapon to be used to attack me is the pharma shill gambit.
Dr. Gorski gets an e-mail
Of all the brands of pseudoscience-supporters who’ve attacked me over the years, none have been as persistent or nasty as the merry band of anti-vaccine activists at Generation Rescue (GR) and its wholly owned subsidiary propaganda blog Age of Autism (AoA). Usually, it’s J.B. Handley who’s been on the attack, beginning two years ago with the usual sort of title J.B. comes up with David Gorski, MD: The Worldwide Wanker of Woo, Dr. David Gorski and His Merry Band of Idiots Don’t Like Full Page Ads, and Dr. David Gorski Jumps the Shark over Desiree Jennings Case. He also apparently hates Steve Novella, too, and there are other anti-vaccine activists out there just as vitriolic, including one named Craig Willoughby, who started a blog, one main purpose of which appears to be to savage yours truly’s good blog buddy, if you know what I mean.
All of this is my usual logorrheic style of introducing my notification that AoA is about to launch yet another such attack. I know this because on Saturday morning I woke up to find this in my e-mail:
Dr. Gorski,
This is Jake Crosby. I am doing a piece about your acknowledgment that disclosure of conflicts of interest is important, yet your lab at Wayne State University stands to benefit from Sanofi-Aventis money for the breast cancer research you are conducting on a drug the company manufactures and markets, Riluzole, which is also being studied for the treatment of autism. Why isn’t any of this disclosed on your blogs? I await your reply.
Sincerely,
Jake Crosby
Age of Autism
Contributing Editor with Autism
http://www.ageofautism.com/jake-crosby/
Oh, dear. It would appear that Jake has found a “gotcha.” At least, he clearly thinks he does. He is sadly mistaken, which is why I sent back an e-mail saying that I couldn’t respond on Saturday with anything other than a brief message pointing out that I receive no money from Sanofi-Aventis or any other pharmaceutical company, nor am I likely to any time in the foreseeable future. Then on Sunday I wrote this post.
In any case, clearly, Jake didn’t bother to read my disclaimer:
Dr. Gorski has been funded over the last decade by institutional funds, the Department of Defense, the National Cancer Institute, the ASCO Foundation, and the Breast Cancer Research Foundation. He currently receives no funding from pharmaceutical companies, although he did once receive a modest payment for an invention from such a company back in the mid-1990s. Indeed, so bereft of pharmaceutical funding is Dr. Gorski that before his talks, when he is required to make his disclosures of conflicts of interest, he often jokes that no pharmaceutical company is interested enough in his research to want to give him any money. Maybe one day that will change, but for now, like most biomedical scientists in academia, he must beg the NIH and other granting agencies for the money to keep his lab going.
It’s true, too. Being a pharma shill apparently just doesn’t pay, although being an Associate Professor of Surgery at Wayne State University at least pays well enough to live on. Depressingly, it’s nowhere near as comfy as sitting back in my underwear, sipping coffee and typing screeds against quacks, cranks, and pseudoscientists. I actually have to work for a living! The indignity! You’d think the pharma overlords who rule the world with David Icke’s reptilians, the Illuminati, the Masons, and the Bilderberg Group could at least slip me a few hundred thousand every now and then to keep my lab churning out reductionist, non-wholistic “Western” science (or at least spring for a really good dinner at The Rattlesnake Club or Michael Symon’s Roast from time to time — now that would be worth selling my scientific soul for). But no. Not even a ride in their famed Black Helicopter (just ask David Ayoub), leaving me feeling as dissed and ignored as Mark Crislip. I continue to submit grant application after grant application to the NIH and various other funding agencies in a very hostile funding environment, hoping against hope to keep my lab going but suffering rejection after rejection. Where’s that filthy lucre? I’m tired and beat, with a severely bruised scientific ego right now, having just recently gotten discouraging news on my latest grant application.
Sanofi-Aventis and your fellow Pharma Lords, take me away!
But I digress. Clearly, a preemptive strike is indicated — nay, demanded against Jake’s impending attack. After all, the best defense is a good offense. However, something still gave me pause. I wondered whether I should bother to respond or, if I respond, it would be perceived as being too “mean” to beat up on Jake, particularly in the wake of a post I did a couple of weeks ago about Julie Obradovic. After all, Jake is young and inexperienced, a college student who’s “on the spectrum,” and it might initially appear unseemly to be too hard on him. Heck, although I’m sure he doesn’t believe it, I actually even kind of like Jake. He seems like basically an otherwise decent kid who’s clearly fallen in with the wrong crowd, namely J.B. Handley’s band of merry anti-vaccinationists at Generation Rescue and Age of Autism. Sadly, they’re corrupting him. Whenever I see Jake do stuff like this, I feel a fatherly impulse to try to set him straight, given that I am actually old enough to be his father. I want to sit him down and calmly ask him in the most fatherly manner I can muster what the hell he’s thinking doing stuff like this and hanging out at AoA promoting pseudoscience. My desire to try to correct a young man’s descent into pseudoscience aside, though, Jake is now an adult, and as an adult he needs to face the consequences of his words and deeds.
Jake Crosby: A rather confused young man with a penchant for conspiracy-laden ad hominem attacks
Sadly, from my perspective, Jake’s words and deeds do require some consequences. Since he’s joined AoA, Jake has made the conscious decision to adopt “poisoning the well”-style ad hominem attacks against anyone whom he perceives as opposing the scientifically discredited notion that vaccines cause autism as his personal modus operandi, apparently having learned at the feet of masters of the ad hominem, such as J.B. Handley, David Kirby, and Dan Olmsted. For example, last year Jake wrote an amazingly conspiracy-filled attack on Adam Bly, founder and CEO of Seed Magazine and creator, of course, ScienceBlogs, which is another home to Peter Lipson and one other SBM blogger, he who must not be named, if you know what I mean (and I think that most of you do).
The attack came in two parts. The first part was “Science” Blogs: Seed Media’s Aggressive Weed, Part I: Fertilizer From Pharma. In it, Jake insinuated that, because Seed and ScienceBlogs accept ads from pharmaceutical companies, both are clearly propaganda outlets for the pharmaceutical industry. Most hilariously, though, Jake claimed that, because Adam Bly had “served at the age of sixteen as the youngest guest researcher at the National Research Council — a Canadian government body that overseas scientific progress, studying ‘cell adhesion and cancer,’” and because of corporate sponsorship of the NRC by Sanofi-Aventis (damn, them, they have their tendrils everywhere!), apparently Adam Bly had been indoctrinated into the Dark Order of Pharma at the tender age of 16 and turned through some sort of dark Satanic ritual into a willing pawn doing the bidding of his pharma masters ever since. (Trust me, I exaggerate only slightly for dramatic effect. Or don’t trust me and read Jake’s original post for yourself.) Then in part II, Jake argues that, because there are members of Seed’s board of directors who have also either worked for or consulted for pharmaceutical companies, Seed and ScienceBlogs must exist only to promote big pharma propaganda, while conflating Steve Novella (who does not blog for ScienceBlogs) and the Discover blog network with ScienceBlogs as though they had anything to do with each other whatsoever.
Since then, Jake has become a very skilled young Padawan who is being corrupted by the Sith and developing into a master of the pharma shill (or the closely related “minion of the CDC”) gambit. He’s applied it to promoters of science over anti-vaccine pseudoscience as varied as Chris Mooney (author of The Republican War on Science, Storm World, and Unscientific America, as well as the journalist who wrote an excellent article on the anti-vaccine movement about a year ago entitled Why does the vaccine/autism controversy live on?); Amy Wallace (who wrote the excellent WIRED article An Epidemic of Fear: How Panicked Parents Skipping Shots Endangers Us All, which resulted in misogynistic attacks on her by J.B. Handley); Trine Tsouderos (the Chicago Tribune journalist who has written multiple excellent articles on vaccine-autism pseudoscience and quackery over the last year or so); the producers of the FRONTLINE documentary on the anti-vaccine movement (SBM commentary here); and, of course, Paul Offit. Perhaps the most tortured use of the pharma shill gambit by Jake occurred when he tried to paint New York Times journalist Gardiner Harris as a hopeless shill for big pharma because — get this — Harris’ brother once worked for pharmaceutical companies (as recently as November 2004, yet!) and now works for a device manufacturer, meaning, as Jake put it, “Gardiner Harris’ brother still deals with pharmaceutical companies.” I kid you not. You just can’t make stuff like this up. (At least I can’t.)
So, with that background in mind, I asked my fellow SBM bloggers whether I should respond, ignore this, or hold back. The unanimous (at least among those who responded to my e-mail) consensus?
Go, get ‘im! So I will.
A retort and explanation from your humble blogger–with science!
Let me just say this to Jake: I do not receive any money from Sanofi-Aventis (or any other pharmaceutical company) to fund my research (or my blogging, for that matter). If you had bothered to read my disclaimer you’d know that. Sadly, I’m not likely to be getting any of that filthy lucre from Sanofi-Aventis any time soon. Again, read my disclaimer. My lab work is entirely funded by grants I’ve gotten through competitive applications and institutional start up funds I received when I accepted my current position. Indeed, Sanofi-Aventis doesn’t even supply me with the drug Rilutek (the trade name of Riluzole) to use for my clinical trial or experiments using mouse tumor models. That’s all funded by grants and startup funds too. Moreover, prior to Jake’s e-mail I had been completely unaware that Riluzole had even been considered for, much less tried as, a treatment for autism. I’m not a neurologist or pediatrician; so it’s unlikely that I would know about that, although I can say, knowing what I know about Riluzole, that it doesn’t strike me as a particularly promising approach.
At this point, no doubt my readers want to know just what on earth Jake is talking about as he tries to paint me as a willing sycophant, toady, and lackey of Sanofi-Aventis; so let me explain. I normally don’t discuss my own research very much on SBM, other than relatively vaguely, because to do so just seems entirely too self-serving to me. (Maybe my lack of self-promotion skills is one reason why the Dark Lords of Pharma want no part of me; or maybe it’s some of my posts, like the one likening vertebroplasty to acupuncture or this one lamenting a pharmaceutical company paying a publisher to publish a fake journal.) However, so that you and Jake understand why his approach is utter nonsense, I have to provide you with a brief primer of my work.
The reason Jake mentioned Riluzole is because it’s one drug I study as a potential therapy for breast cancer. In fact, if I do say so myself, the project looking at Riluzole in breast cancer is an example of why science-based medicine is so cool compared to the fairy dust of “alternative” medicine or the pseudoscience that is the anti-vaccine movement. It all began a while back with a basic scientist at Rutgers named Suzie Chen, who wanted to make a transgenic mouse to study adipogenesis. So she used standard techniques to insert a gene she wanted to study, but after the transgenic mice were born the mice in one of the strains all developed skin lesions that looked like melanoma by the age of two to four months. Being a good and careful scientist, Dr. Chen realize that she had inadvertently created something really interesting, a mouse model of melanoma. So, teaming up with Dr. James Goydos, my surgical partner at The Cancer Institute of New Jersey (which is where I was at the time), Dr. Chen went back and looked at what had happened and figured out that the transgene had disrupted part of the regulatory region of a gene known as Grm1 (GRM1 in the human), which codes for a gene known as metabotropic glutamate receptor-1 (mGluR1), in such a manner that mGluR1 was being made at a level far higher than normal.
A word of explanation is in order here. Glutamate is a major neurotransmitter in mammals that functions by binding to either ionotropic or metabotropic receptors (genes: GRM1-GRM8; receptors: mGluR1-mGluR8). I’m not going to talk about ionotropic glutamate receptors. mGluRs belong to a family of G-protein-coupled seven transmembrane domain receptors (GPCRs), which mediate responses to a diverse array of signaling molecules, including hormones, neurotransmitters, chemokines, and autocrine and paracrine factors. In the mammalian CNS, mGluRs, which are categorized into either group I, II, or III receptors based on sequence, what compounds they bind, and how they signal in the cell, are essential for normal neuronal function, and have been implicated in diverse neurological pathology, particularly ALS. Their existence and importance in melanoma had previously been unsuspected.
Again, being a careful scientist, Suzie Chen went back and made a new transgenic mouse, this time using Grm1 engineered in such a way that it is only expressed in the melanocytes of the mouse strain created and observed very close to the same thing. Now here’s where things get really interesting. It turns out that there is already an FDA-approved drug that blocks glutamate release and signaling. This drug is Riluzole, and it’s used to treat amyotropic lateral sclerosis (ALS, a.k.a. Lou Gehrig’s disease). In fact (and Steve will correct me if I’m wrong) Riluzole is the only currently known drug that prolongs the life of ALS patients, albeit not by a lot. Truly, this was serendipity of the sort that most scientists never see in their lifetimes.
Why is Riluzole so important in this story, even though it doesn’t specifically inhibit the activity of mGluR1? Because it was already FDA-approved. Thus, testing it in cancer didn’t require a new drug approval, only an approval to test the off-label use of Riluzole against melanoma, which is what Drs. Goydos and Chen did, demonstrating that Riluzole inhibits the growth of melanoma in cell culture and in mouse tumor models, that it is an oncogene in melanoma, and that it showed promise against melanoma in a small pilot clinical trial. Dr. Goydos is currently running a clinical trial to test Riluzole in melanoma patients, one for patients with unresectable melanoma.
In the meantime, about three years ago I started collaborating with Dr. Goydos to see if mGluR1 is expressed in breast cancer. It turns out that it is and that it is active. Indeed, our first paper reporting these results has been submitted, and I’m busily putting together a paper to report our results in animal models, results that I reported at the recent ASCO meeting. I also have a small pilot clinical trial, funded by The ASCO Foundation and the Breast Cancer Research Foundation, that is going on right now. In the meantime, while my collaborators keep working on melanoma and some other tumor types, I’m working on figuring out how mGluR1 functions in breast cancer with the hope that the knowledge will lead to a treatment for breast cancer that can be taken as a pill and that has very low toxicity. Riluzole meets both criteria. If it passes clinical trials, it may well be a very useful drug for potentiating the effects of other cancer therapies, such as chemotherapy and radiation.
If this research works out, I’d like to think I did something good, building on the work of Drs. Chen and Goydos, to benefit breast cancer patients. In fact, I’m quite excited that we’re on the verge of publishing our first paper on our work and currently deep into the process of writing our second paper. Both, I hope, will appear before the end of 2010.
Real conflicts of interest versus phantom conflicts of interest
The bottom line is that, for Jake’s insinuation to represent a true COI that needs to be reported up front when I write about vaccines and autism, at least three things would have to be true. First, I would have to be receiving money from Sanofi-Aventis. I am not. Second, I would have to have the reasonable expectation to receive money from Sanofi-Aventis. I do not. I’m not even angling for money from Sanofi-Aventis to run my lab. Third, I would have to know that Riluzole is being tested as a treatment for autistic children. I did not until Jake wrote to me. In fact, Jake isn’t even correct. I searched ClinicalTrials.gov, and all I could find was a study called Riluzole to Treat Child and Adolescent Obsessive-Compulsive Disorder With or Without Autism Spectrum Disorders. It’s clear that the primary purpose of this trial is to test Riluzole in children with obsessive-compulsive disorder and they just happen to be including children with ASD as well. I really don’t know why; it seems to me that including children with ASDs would only make the results of the study more difficult to interpret.
In any case, even if I had known that, it still wouldn’t have been a COI. I’m not a neurologist, and I don’t treat ASD or OCD. I’m never going to be doing research with Riluzole in children with ASD, OCD, or both. I’ll attempt to follow Jake’s tortured logic. Apparently, because Jake buys into the AoA-promoted myth that autism is in reality due to “vaccine injury,” to him anything that is used to treat autism is in fact equivalent to treating “vaccine injury.” Apparently it follows that, because I’m doing research involving Riluzole and there is a clinical trial using Riluzole to treat children with OCD, some of whom will also have an ASD, I must be defending vaccines and arguing against the idea that vaccines cause autism because I’m heavily invested in Riluzole as a treatment for “vaccine injury” as well as cancer. Either that, or Sanofi-Aventis must be paying me big bucks or because I stand to make money off of Riluzole if it’s found to be a good treatment for OCD and/or breast cancer.
I must admit, Jake’s “logic” confuses me. But, then, I don’t see conspiracies everywhere. (Well, most of the time, anyway.)
As I come to the end of one of my typically logorrheic magnum opi, I wonder two things. First, I wonder if anyone’s reading anymore or if everyone’s gone to sleep. Second, and more importantly, I wonder once again if I’ve been too hard on Jake. I don’t think so. If Jake did not have a documented history stretching well into last year of trying to slime everyone he can who has the temerity to argue against the scientifically discredited idea that vaccines cause autism or to criticize GR and AoA for promoting that myth with charges of either being in the pay (and therefore thrall) of big pharma, I probably would have written a shorter and friendlier version of this post and sent it to Jake as a private e-mail. Unfortunately, Jake does have a history of trying to slime anyone he thinks he can dig some “dirt” on with the charge of an undisclosed COI or of being in the thrall of big pharma and clearly has me in his sights as his next target. Sadly, Jake has learned from J.B. Handley and others at AoA.
Jake’s conspiracy-laden posts finding tenuous connections between his enemies that he trumpets as “undisclosed COIs” that to him apparently completely invalidate the science being argued, indicate to me that Jake fancies himself an investigative journalist and wants to play in the big leagues. Unfortunately, as I pointed out before, dismissing an opponent’s argument solely on the basis of a COI, perceived, real, or imaginary, is intellectually lazy. Clearly, Jake isn’t even ready to play Class A ball yet. He’s probably out of his depth in Little League. Even more unfortunately, as long as Jake stays affiliated with AoA, I don’t think he ever will be ready for the big leagues. It’s a shame, because Jake seems like a smart and ambitious kid who could have an impressive career at something and actually contribute to society rather than spreading conspiracy theories. What a waste. Maybe one of Jake’s professors will take an interest in him, take him under his or her wing, and mentor him away from the craziness.
I can always hope, can’t I?

   Demonizing “Big Pharma”
Steven Novella
Link
To be blunt up front – SBM is not apologetic about the pharmaceutical industry. We get zero funding from any company, and have no ties of any kind to “big pharma.” In today’s world I have to spend time making that clear, because despite the reality critics are free to assume and falsely claim that our message is coming straight from the bowels of hell (a.k.a. the pharmaceutical industry).
We promote science-based medicine and criticize pharmaceutical companies along with everyone else when they place other concerns ahead of scientific validity, or promote bad science, for whatever reason.
It has become fashionable, however, to not only criticize the pharmaceutical industry but to demonize them – and the term “big pharma” has come to represent this demonization. Cynicism is a cheap imitation of skepticism – it is the assumption of the worst, without careful thought or any hint of fairness.
A recent article by Martha Rosenberg is an excellent representation of the mindless demonization of the pharmaceutical industry – good for scoring cheap points, but very counterproductive. She essentially accuses big pharma of inventing diseases in order to sell their products.
The premise strikes me as profoundly naive – which diagnostic entities are considered legitimate diseases is actually a complex question that is debated within the medical field. Rosenberg acts as if diseases can be invented out of whole cloth and then imposed upon medicine by a pharmaceutical executive. It is a grand-conspiracy type of thinking which erodes under scrutiny.
After hinting at anti-vaccine leanings, she writes:
Now pharma is back to creating new diseases, patients, risks and “awareness campaigns” faster than you can say thimerosal (the vaccine preservative that started the backlash.)
No – thimerosal did not start the backlash, Andrew Wakefield demonstrably did, with the MMR vaccine that never contained thimerosal. Thimerosal was simply act 2, after the evidence failed to find a link between MMR and autism (and of course there is also no link between thimerosal and autism either). But Rosenberg acts as if the anti-vaccine movement is a justified backlash against the excesses of big pharma – nice historical revisionism.
The sad fact is, Rosenberg might have a kernel of a legitimate point if she did not come across with her anti-scientific conspiracy mongering. That is why such demonization is so counterproductive – it actually backfires and let’s pharmaceutical companies off the hook for their real excesses.
Harriet Hall, for example, wrote an excellent piece on osteopenia – (Osteoporosis Drugs: Good Medicine or Big Pharma Scam?) which takes a properly nuanced and balanced approach to such questions. Do we really need to be treating pre-osteoporosis? The evidence should ultimately guide us. What pharma is guilty of doing is jumping prematurely on the bandwagon of a questionable diagnosis because it is a new market for them.
I think the same is true of the drugs that are now approved for the treatment of fibromyalgia – a controversial diagnosis, to say the least. But here we see more complexity and nuance. The FDA requires that a drug be indicated to treat a disease – not a syndrome or symptom. So there is no drug indicated for treating neuropathic pain as a symptom – drugs have to be indicated for diabetic neuropathy or post-herpetic neuralgia.
This forces pharmaceutical companies to find a disease, even when they have a drug that can potentially alleviate a symptom. Fibromyalgia is the perfect example of this – the very diagnosis itself is mostly used as a garbage pail diagnosis for vague syndromes of muscle pain and tenderness with fatigue and poor sleep. But you cannot get FDA approval to treat vague muscle pain.
Meanwhile, doctors are struggling to understand these syndromes and come up with a proper system of labeling what we find. We don’t want to prematurely use the “disease” label, but we also need to recognize patterns of patient complaints. I prefer terms like “myofascial pain syndrome” because it says what it is without implying a specific disease.
But regulation exists in its own world, and the FDA demands a disease label. So we have drugs, which are likely fine for the symptomatic treatment of myofascial pain, indicated for a dubious diagnosis (at least as it is often applied) like fibromyalgia. But it is doctors that invented the concept of fibromyalgia, and we still debate about it.
Rosenberg, however, cuts through all this nuance and goes for the simplistic and cynical conspiracy theory – pharma “invented” fibromyalgia to sell its drugs. She writes:
Nothing proves pharma’s when-the-medication-is-ready credo better than the legions of people who have fibromyalgia now that Cymbalta, Savella and Lyrica are available to treat it.
This is more historical revisionism. Having lived, and practiced medicine, through the fibromyalgia controversy it is clear that what happened is fibromyalgia became a popular diagnosis for the common vague syndrome I described above. Much after fibromyalgia became a popular diagnosis, some pharmaceutical companies saw it as a potential market. Rosenberg therefore has it backwards.
What we do have to recognize is that, now that there are drugs indicated for fibromyalgia, those pharmaceutical companies that make those drugs are invested in the reality and popularity of the diagnosis. They may therefore seek to distort the debate in that direction.
Rosenberg also embarrasses herself by criticizing the notion that there is an epidemic of sleep disorders in our society – the evidence suggests that there is, and it is under-treated. She further goes after adult ADHD and adult autism. The alternative is that autism and ADHD are childhood diseases only and always spontaneously resolve by adulthood – a scientifically untenable, and even laughable, position.
She further completely distorts the notion of “treatment resistant” conditions. She misinterprets that fact that many drugs are initially approved for adjunctive (add-on) therapy. This is not because the notion of “treatment resistance” was invented by big pharma. It is partly due to the fact that it is easier to do clinical trials where a new treatment is added to an established treatment, rather than to prove equivalence as stand alone therapy. So pharmaceutical companies go after the low-hanging fruit to maximize their return on investment.
Also – some patients are difficult to treat, and when one approach is not adequate it is nice to have more options. Rosenberg somehow turns this into a negative.
Conclusion
Rosenberg’s approach to this complex issue is simplistic, naive, and conspiracy-mongering. She brings no useful insight to the discussion. She also demonstrates nicely the method of “demonizing” a convenient target – re-write history, white wash over all complexity and nuance, and cast everything into a maximally sinister light.
But further Rosenberg shows that taking such an approach is highly counterproductive. The pharmaceutical industry, like every industry, needs an effective watchdog to guard against abuse and excess. I also think they require thoughtful and effective regulation (although this question is difficult to disentangle from political ideology).
Rosenberg and other big pharma conspiracy theorists make ineffective watchdogs and critics, because their criticisms are paper thin and easily countered . By not recognizing the complexity of the issues involved, or making any attempt at fairness, Rosenberg is easily dismissed.
If I were a conspiracy nut I might even suspect that people like Rosenberg are actually fronts for big pharma, used to create a straw man of criticism that they can then easily knock down to show that all criticism is weak and invalid.

   Crank “scientific” conferences: A parody of science-based medicine that can deceive even reputable scientists and institutions
David Gorski 
Link
If there’s one thing that purveyors of pseudoscientific medical modalities crave, probably above all else, it’s legitimacy. They want to be taken seriously as Real Scientists. Of course, my usual reaction to this desire is to point out that anyone can be take seriously as a real scientist if he is able to do science and that science actually shows that there is something to his claims. In other words, do his hypotheses make testable predictions, and does testing these predictions fail to falsify his hypotheses? That’s what it takes, but advocates of so-called “complementary and alternative medicine” (CAM) or “integrative medicine” (IM, or, as I like to refer to it: “integrating” quackery with scientific medicine) want their woo to be considered science without actually doing the hard work of science.
There are several strategies that pseudoscientists use to give their beliefs the appearance of science, a patina of “science-y” camouflage, if you will. One, of course, is the cooptation and corruption of the language of science, which has been a frequent topic on this blog, particularly in posts written by Drs. Atwood and Sampson. Another is to produce journals that appear to be science, but are anything but. I’ve discussed one example, the Journal of American Physicians and Surgeons and Medical Acupuncture, but others include Homeopathy, the Journal of Alternative and Complementary Medicine, and Medical Hypotheses, which recently was forced to retract a horrible paper by arch-HIV/AIDS denialist Peter Duesberg. What’s worse is that some of these journals are even published by what are considered major publishers, such as Mary Ann Liebert, Inc., and Elsevier.
There is, however, a third strategy. How do scientists communicate their findings to other scientists, as well as meeting and mingling with other scientists? Why, they hold scientific meetings, of course! These meetings can be small or even as large as the American Association for Cancer Research meeting, which is attended by around 15,000 cancer researchers each year. So, too, do cranks hold meetings. These meetings often have all the trappings of scientific meetings, with plenary sessions, smaller parallel sessions, poster sesssions, and an exhibition hall, complete with exhibits by sponsoring companies. Sometimes these meetings can even appear so much like the real thing that they take in legitimate researchers and legitimate universities. Here, I present two examples of such conferences.
THE UNIVERSITY OF TORONTO: DUPE OR WILLING ACCOMPLICE?
The first such conference that comes to mind when I think of fake scientific conferences is the autism quackfest known as Autism One. Autism One is a conference of believers in two things: (1) that vaccines cause autism and (2) that “biomedical” and CAM/IM therapies can treat and even reverse autism. This conference, heavily promoted on anti-vaccine websites like Age of Autism, is held every year in Chicago near or on Memorial Day weekend. It is generally held in a hotel near O’Hare International Airport (downtown locations must be too expensive), and in general features the “luminaries” of the vaccines-cause-autism underground giving talks on the “science” of the vaccine-autism link and all manner of “biomedical” interventions, the vast majority of which are, in my opinion, pure quackery. For instance, the 2008 Autism One conference featured Boyd Haley, the father-son team of autism woo-meisters Mark and David Geier, Mayer Eisenstein, and a number of others. Examples of the talks at that conference included “Over 50 years of known toxicity! (Unsafe at any Concentration)” by Frank Engley, who said this: “NOT GRAM- NOT MILLIGRAM – NOT MICROGRAM- BUT NANOGRAM!” Its keynote speaker was–surprise! surprise!–Jenny McCarthy, whose ignorance of science is beyond belief and has led her to become, in essence, the celebrity face of the anti-vaccine movement, even leading marches on Washington against vaccines.
In 2009, Autism One made a bit of a splash, but not in a good way. Basically, what happened is that the Chicago Tribune noticed the Geiers’ Lupron protocol for autism. Specifically, Mark and David Geier have been subjecting autistic children to a drug that is a powerful suppressor of sex hormone production because they actually think that testosterone somehow binds mercury and that lowering testosterone concentrations will make chelation therapy work better. Truly, this is some of the most despicably off the wall woo that I’ve ever encountered. it combines bad biochemistry with anti-vaccination pseudoscience of combining testosterone-lowering treatments with the dangerous quackery that is chelation therapy for autism. Then there was Dr. Mayer Eisenstein, founder of the crunchy and “alternative” HomeFirst practice. This is the same guy who said that he had never seen a case of autism in an unvaccinated child in his practice, a case of data-free selective memory so blatant that later the story morphed into his practice having a huge computer database to back up his claim. He is a master of anti-vaccine rhetoric, including his beliefs that vaccines cause autism, that the government is covering it up, that the American Academy of Pediatrics is hopelessly in the thrall of big pharma, and that “every doctor now essentially in this country has done something as heinous as the Nazis did, unknowingly” because they give vaccines to children.
And, of course, Jenny McCarthy was the keynote speaker in 2008 and then again in 2009. Any one want to make any bets against her being the keynote speaker for Autism One 2010? I don’t.
As annoying as Autism One in Chicago is, however, there is one thing that very obviously distinguishes it from real scientific conferences, specifically the lack of a university or academic organization sponsoring it. It’s held out in a hotel near the airport, far from downtown Chicago, and there is very little pretense of academic endeavor. The patina of a scientific conference is thin indeed.
I wish I could say the same thing about an offshoot of Autism One, namely Autism One Canada:
If you live in Canada, the Northeastern United States, the Great Lakes area, or beyond, you will not want to miss this conference from Autism One, Autism Canada, and the University of Toronto’s Dalla Lana School of Public Health. The 2-day main conference will be held Saturday, October 31 and Sunday, November 1, preceded by a 1-day training program for practitioners on Friday, October 30. This conference features two dozen of the most highly respected names in the autism community and provides the most up-to-date information to help your child. From implementing the best biomedical treatments to realizing the benefits of educational therapies to adolescence and adulthood issues, the conference brings you the answers to be your child’s most effective healer and powerful advocate.
The flier then advertises that it is sponsored by the SickKids Foundation, Spectrum Supplements, and Ontario Hyperbaric Oxygen Therapy Centre Inc. A scan of the actual pamphlet advertising the conference can be found below:

Autism One Canada doesn’t appear quite as bad as the Mothership Autism One in Chicago usually is. At least, “scientists” like the Geiers don’t appear to be speaking there. But there’s still plenty of dubious science there to go around, starting with the keynote speaker, Dr. Martha Herbert. She shares one characteristic with at least a couple of “luminaries” of the anti-vaccine “biomedical” movement. As Autism Diva and Kevin Leitch pointed out, like Dr. Mark Geier and Dr. Boyd Haley, Dr. Herbert has been slapped down by the courts, which used the Daubert standard to reject her testimony claiming that a child was made autistic by a reaction to mold growing in the condo she was living in. The court found:
Dr. Herbert’s publications indicate that she is an outspoken advocate of increased attention to the possibility of environmental influences. Even she, however, despite that acknowledged perspective, speaks in her published work of possibilities and potentialities, rather than of the ‘reasonable degree of medical certainty’ to which she offers to testify under oath in this case.10 Neither Dr. Herbert’s publications, nor any others cited, identify mold exposure as even a suspected, still less a known or proven, trigger of autism……Dr. Herbert’s method, to the extent the Court can discern it from the materials offered, is a series of deductions based on possibilities…..*Clearly, Dr. Herbert’s method is not generally accepted in the scientific community*. Dr. Herbert’s theory of environmental triggers of autism may some day prove true. It has not yet. Her proffered testimony does not meet the standard of reliability required by the case law, and cannot be admitted in evidence at trial.
Dr. Herbert is, it turns out, a big fan of the idea that autism has something to do with neuroinflammation. Unfortunately, none of her publications persuasively presents evidence for this hypothesis, and lately she’s publishing in bottom-feeding alternative medicine journals articles with titles like Learning From the Autism Catastrophe: Key Leverage Points. Suffice it to say, Dr. Herbert is big on “biomedical” woo, so much so that anti-vaccine propagandist David Kirby likes to cite her and Age of Autism loves her. Unfortunately, for all her grandiose claims that neuroinflammation is a major cause of autism and that mold and other environmental influences trigger it, Dr. Herbert’s publication record does not support these assertions. Go ahead. Head over to PubMed and look at Dr. Herbert’s publication record. I’ll wait. She has listed 15 publications about autism, of which:
six are review articles
two are in alt-med journals, and one of these is an interview
one is a paper with dozens of authors reporting the results of mapping autism risk loci using genetic linkage and chromosomal rearrangements. (Dr. Herbert is solidly right in the middle of the huge pack of authors.)
Of the remainder, Dr. Herbert only appears to be first author or senior author on four publications on autism containing original research, and these appear to be all imaging studies of the brains of autistic children. In other words, Dr. Herbert is making claims far beyond what her publication record in the peer-reviewed literature can, even under the most charitable interpretation possible, support. Nothing in her publication record appears to support her concept of autism being a systemic, rather than brain-based condition. There’s nothing there to support a link between autism and gut disorders; nothing to support a link between autism and immune dysfunction; and nothing to support a link between “environmental influences” and autism. That’s not to say that there aren’t environmental factors that influence the development of autism; it’s just that there’s nothing in Dr. Herbert’s publication record to support such a hypothesis or to identify what, if anything, those environmental factors might be.
This is the “keynote speaker.” She is, however, faculty at Harvard University.
The other speakers aren’t much better. I’ll admit, the woo-to-normal ratio isn’t as high as it was in Chicago last May, but it’s still pretty high. It’s just that this is mostly second tier “biomedical” stuff. It’s the second stringers, for the most part. For instance, there’s Dan Rossignol, who’s big on hyperbaric oxygen as a treatment for autism with very little data. There’s Amy Yasko, who’s known for somehow thinking that one can treat autism (and lots of other disorders) with orally administered RNA solutions. I work with RNA all the time. It’s damned unstable. It would not survive even the saliva without being broken down into its components. Actually, RNA won’t survive long in aqueous solution without being frozen at at least -20° C, preferably -80° C. Yet this Dr. Yasko claims that “we have found that by using highly specific solutions of RNAs we are able to enhance our cells natural ability to communicate with one another.”
I’d call Yasko’s woo homeopathic RNA therapy, except that apparently there are real yeast nucleotides in her concoction. Speaking of homeopathy, there’s a real homeopath there called Rudi Verspoor, HD(RHom), DMH, whose presentation is described thusly:
Autism spectrum disorder (ASD) is a complex disorder with multiple causes, and each case presents its own unique and distinct challenges. The key is to treat the causes and not just the effects. Treating symptoms may provide some improvement but does not provide a permanent solution. Heilkunst, which includes homeopathy, is a comprehensive approach to addressing the multiple deeper underlying causes of a case in a given sequence over time, while using energetic, homeopathic medicines. It provides a clear map of the problem and of how to get out of the swamp of autism spectrum disorders that is safe, non-toxic, and effective even in complex seemingly non-responsive cases.
Well, two out of three ain’t bad, I guess. There’s no doubt that homeopathy is safe and nontoxic. After all, it’s water. But effective? Not so much. Come to think of it, I take back what I said about Autism One Canada not being as bad as Autism One Chicago. Advocating homeopathy definitely ups the ante to anything the mothership could come up with. But other forms of woo rule too: For instance, there’s Bryan Jepson, who comes straight from Andrew Wakefield’s Thoughtful House. His talk will be:
Treating Autism: Understanding Biomedical treatment options
Dr. Jepson will describe the multiple organ system involvement of the autism spectrum including the brain, the gut, the immune system and the detoxification/metabolic system. He will discuss the mechanisms for disease in each of these areas and then talk about treatment options that are being used by clinicians in order to try to correct the underlying biochemical abnormalities that are contributing to the behavioral presentations of the disorder.
Whenever you hear anything about “detoxification” and “biomedical” in the same talk, run, don’t walk. You’re almost certainly hearing only the finest anti-vaccine and environmental woo. If the speaker is someone like Dr. Jepson, who is currently working with the Dark Lord of the Anti-vaccine Movement himself, the man who, while on the payroll of a lawyer who sues vaccine companies, produced an incompetently done and fraudulent study that launched a thousand anti-vaccine quacks in the U.K., in the process producing a scare that the MMR causes autism so great that vaccination rates fell to the point where the measles has become pandemic again, you’re certain to hear anti-vaccine nonsense.
Finally, let’s not forget that Dr. Bob Sears will be speaking there, too. Dr. Bob craves respectability and wants his “alternative” vaccination schedule to be taken seriously as a viable alternative, but he can’t marshal one bit of science to support his beliefs. Worse, lately he’s been writing for Age of Autism and that repository of all things anti-vaccine, The Huffington Post. I refer readers to Dr. Snyder’s excellent post Cashing In On Fear: The Danger of Dr. Sears for a more detailed deconstruction of his “alternative” vaccine schedule and why he has gravitated closer to the anti-vaccine movement.
When I first heard of Autism One Canada, I wondered if this was a case of Autism One and Autism Canada simply renting space at the University of Toronto and trying to coopt its good name, much as creationists tried to coopt the name of the Smithsonian by showing The Privileged Planet there. I’d like to hope that it’s the same sort of deal at the University of Toronto, where such events have to be listed as “co-sponsored” by the host institution in order for the event to occur there. I’ve searched the Dalla Lana School of Public Health at the University of Toronto and haven’t been able to find a single mention of this conference. Alternatively, what may have happened is that someone at the University of Toronto confused anti-vaccine pseudoscience for “cutting edge” and somehow was thus fooled into cosponsoring this conference. What it is is half and half.
First the good half. A reader at my other blog has informed me that the Dana Lana School of Public Health has completely disavowed any involvement in the conference other than that one of its junior faculty, Dr. Marianna Ofner, Assistant Professor of Epidemiology, is giving a talk there, entitled The Epidemic of Autism in Canada, which does not sound encouraging, as it sounds as though she may buy into the concept of an “autism epidemic” so beloved by the anti-vaccine movement. Be that as it may, it’s unclear why or how Autism One Canada thought it could get away with listing the Dalla Lana School of Public Health as one if its sponsors.
Now the bad half. A number of people, myself included, upon learning that the SickKids Foundation was listed as having sponsored Autism One Canada, wrote e-mails and letters of complaint and concern. They received in response a canned reply from the Grants Officer at SickKids Foundation, Pam Gilliland:
Dear Mr. [suppressed],
Thank you for your email concerning the conference: Changing the Course of Autism in Canada, organized by Autism Canada Foundation in collaboration with the Dalla Lana School of Public Health University of Toronto and Autism One.
The goal of this conference is to provide a respectful forum for parents, therapists, doctors, researchers and individuals with Autism Spectrum Disorder to share, learn and work collaboratively to expand knowledge of treatment interventions for Autism. There will be a cross section of speakers.
As part of SickKids Foundation’s National Grants Program, support is offered for conferences, workshops or symposia which are relevant to the health of Canada’s children. The purpose of the conference grants program is to support events which are organized by and/or for families with children with health challenges.
The review process is competitive and funding is limited, with a maximum of $5,000 per grant. Each conference grant application is assessed in terms of its relevance to the health of Canadian children up to 18 years of age, as well as for its fit with the conference grants program goals.
The Foundation sees value on information sharing between medical staff, community organizations, and families. It is important for families to have opportunities for open dialogue with health professionals in order to get an understanding of current research and practices.
The Foundation takes a neutral stance on complementary and alternative health care. We actually have a history of funding research on complementary and alternative health care for paediatrics. The use of complementary and alternative health care products and therapies are on the rise across Canada and there is little research on the safety and efficacy of many of these treatments and products for children and youth, as well as the effects of the interactions between natural health products and conventional medicine. For this reason, the Foundation has taken a first step to build research capacity on which to base practice and policy in these areas.
To this end SickKids Foundation funded Autism Canada in the amount of $5,000 to support their conference: Changing the Course of Autism in Canada.
I hope this helps to answer some of your questions.
Best regards,
Pam Gilliland
Grants Officer, National Grants
SickKids Foundation
525 University Avenue, 14th Floor
Toronto, Ontario M5G 2L3
(P) 416.813.6166 x 2354
(F) 416.813.7311
Pamela.Gilliland@sickkidsfoundation.com
http://www.sickkidsfoundation.com/nationalgrants
Depressing. Regular readers of this blog will recognize this e-mail as worthy of the Weekly Waluation of the Weasel Words of Woo (W5) What bothered me about Ms. Gilliland’s e-mail even more than what she wrote is what Ms. Gilliland left unwritten. Despite several of the letters of complaint mentioning the anti-vaccine slant of Autism One, Ms. Gilliland wrote not a word about vaccines, as if she were assiduously and intentionally ignoring this very important point, namely that by funding Autism One Canada, the SickKids Foundation is lending its name to one of the greatest threats to the health of children: the anti-vaccine movement. If the anti-vaccine movement gets its way, vaccination rates will fall, and children will suffer and die. I realize that the anti-vaccination movement doesn’t see it that way, that its adherents think that vaccines cause autism, but the end result of their efforts will be the suffering and deaths of children nonetheless. Yet, here is a charity dedicated to helping children through scientific research and which funds a large amount of truly good research supporting a group that is part of a movement that is the most serious threat to public health in general and children’s health in particular. I would like to think that Gilliland probably has no clue that this is the case, but her refusal to address the issue implies to me a level of shame, as though she knows what Autism One really stands for.
This is the sort of thing that happens when pseudoscience infiltrates academia.
WHEN SCIENTISTS FAIL TO SEE THE OBVIOUS
The National Vaccine Information Center (NVIC) has a deceptively benign-sounding name. There’s little doubt that this was an intentional choice of its founder, Barbara Loe Fisher, who is profoundly antivaccine, as her blog’s title Vaccine Awakening suggests, as does her defense of Andrew Wakefield, spreads misinformation about the HPV vaccine and brain injury (and it is misinformation), and her promotion of “vaccine choice” (which is very much like the Orwellian term “health freedom” in that it is designed to disguise the true intent behind it. It’s not a coincidence that a well-known “vaccine injury” litigator is on the board of directors of NVIC or that it includes a list of law firms on its website that specialize in “vaccine injury” cases.
The bottom line is that the NVIC is one of the oldest anti-vaccine groups out there, and Barbara Loe Fisher is the grande dame of the anti-vaccine movement, having been at it since the 1980s. A brief perusal of the NVIC website will show that it supports the discredited idea that mercury in vaccines causes autism and the equally discredited notion that vaccines cause autism and all sorts of dire outcomes. Indeed, there’s even a “vaccine memorial” for the “victims” of vaccines.
The NVIC has a conference, too, the 4th International Public Conference on Vaccination in October. Barbara Loe Fisher announces her conference in this YouTube video:
Note the clever P.R. Note how Fisher never once uses the word “against” or “anti-” anything. She paints her antivaccine group NVIC as “pro-education” and “pro-informed consent” and for “defending your right to choose.” She even ends her video by saying, “Stand up for vaccine freedom of choice.” To look at Generation Rescue, note how Jenny McCarthy is always saying she’s not “anti-vaccine” but rather “pro-safe vaccine.” The anti-vaccine crowd knows that it’s far better PR to be seen as being for something rather than just against something. Of course, her spin belies the speaker list:
Andrew Wakefield. Readers of this blog need no introduction to Dr. Wakefield. He’s the guy who started the MMR scare in the U.K. through a combination of bad science, lack of ethics, and falsifying data.
Barbara Loe Fisher. The founder of NVIC. ‘Nuff said.
Joe Mercola, DO. Woo-meister supreme, anti-vaccinationist, spreader of misinformation, and promoter of quackery on his website Mercola.com.
Rick Rollens. One of the strongest believers in the pseudoscience claiming that vaccines cause autism.
Kim Stagliano. Blogs for Age of Autism. Enough said.
Gary Null. Whoa. This guy has it all. He’s a perfect storm of pseudoscience: Anti-vaccine to the core, an HIV/AIDS denialist, and a promoter of all manner of dubious medicine–to put it kindly.
Look who else is speaking, though. That’s right, it’s David Ayoub. Yes, that David Ayoub, whom I like to call David “New World Order” Ayoub. This is a guy who’s so into conspiracy theories that he thinks that vaccines are part of a population control plot by the Illuminati, along with birth control, women’s rights, and all sorts of other nefarious plots by the government, all complete with black helicopters and everything. Really. Just check out this video:
Check it out especially around the 44 minute mark, where Ayoub starts talking about “stealth population control” and genocide. Truly, there is some serious craziness in his talk. I have to wonder if Ayoub will manage to restrain his New World Order/Illuminati/Black Helicopter ramblings for this conference. (No doubt Barbara Loe Fisher told him to check his crazy at the door.) Speaking of Black Helicopters, Ayoub actually does mention them right around the 57 minute mark of the video above.
But, wait, there’s more! Check out these talks:
“Homeopathy: The Impossible Cure”
Amy Lansky, PhD
Executive Board, National Center for Homeopathy
“Pregnancy, Birth & Child Health: Why Chiropractic Works”
Jeanne Ohm, DC
Executive Board, International Chiropractic Pediatric Association
“Vaccines & Inflammation”
Andrew Wakefield, MD
Executive Director, Thoughtful House Center for Children
“Homeopathy & Vaccines: What Our Pets Teach Us”
Richard Pitcairn, DVM, PhD
Founder, Animal Natural Health Center
“Vaccines, Informed Consent & Public Health: A Chiropractic Perspective”
Matthew McCoy, DC,MPH
Associate Professor, Clinical Sciences, LIFE University
Wow! So, there’s not just Andrew Wakefield, the subject of a rather shoddy NBC news report a couple of weeks ago and the man who is more than anyone else responsible for the MMR scare in the U.K., but two talks on homeopathy! What does homeopathy have to do with vaccines, I wonder? Who knows? Yes, I realize that some homeopaths try to liken homeopathy to vaccination as “like cures like,” but that’s a specious analogy. In any case, if you peruse the list further, you’ll see lots of talks on “health freedom” (which I like to call “freedom for quacks”) and how to evade vaccination by using philosophical exemptions. It’s not a coincidence that a well-known “vaccine injury” litigator is on the board of directors of NVIC.
Yet, paradoxically, despite the presence of so many–shall we say?–scientifically dubious speakers, there are a handful of reputable scientists and academics speaking at the NVIC conference For example, there is (or, as we shall see, was) Tom Jefferson, MD, who is a prominent member of the Cochrane Group, being one of the editors of the Cochrane ARI Group, a member of four other review groups and two methods groups, and the coordinator of the Cochrane Vaccines Field. In his capacity with the Cochrane Group, he is also co-author on many of the Cochrane Reviews on influenza and the flu vaccine. So what is Dr. Jefferson doing listed not only as a speaker at the NVIC Conference, but as an award recipient? Indeed, a general e-mail to NVIC attendees forwarded to me by a “mole” states:
The Saturday night special Awards Banquet recognizes the
accomplishments of Andrew Wakefield, who is receiving NVIC’s Humanitarian Award; and presentations of the NVIC Courage in Science Award to Italian epidemiologist Tom Jefferson; the NVIC Visionary Award to holistic health pioneer Joe Mercola; the NVIC Courage in Journalism Award to Mothering magazine founder Peggy O’Mara; and the NVIC Parent Leadership Award to child health advocate Vicky Debold.
As a proponent of science-based medicine, I’d love to be in the same awards ceremony receiving a “Courage in Science” award from the NVIC along with Andrew Wakefield, Joe Mercola, and the editor of Mothering magazine (yes, that Mothering magazine, the same one that promoted Christine Maggiore’s HIV/AIDS denialism and whose contents and discussion boards are filled with the rankest anti-vaccine pseudoscience). (Obviously, that’s sarcasm; what the NVIC considers “courageous” science considers pseudoscience.) So what on earth motivated Dr. Jefferson to accept this invitation? Was he buttered up by the possibility of receiving an award? Was he ignorant about the nature of the NVIC, the nature of which a simple perusal of its website, with its online shrine and memorial to “victims” of vaccination, its Cry for Vaccine Freedom Wall and its approving links to all manner of pseudoscience would have told him in less than a few minutes?
The reasons why Barbara Loe Fisher invited him are obvious, at least to me. Jefferson is well-known as a bit of a gadfly when it comes to the influenza vaccine, having authored a BMJ commentary in 2006 that criticized the evidence for the efficacy of flu vaccines as being of lower quality. This article is widely cited in anti-vaccine circles. Similarly, he authored a systematic review of the literature that concluded that industry influence has an undue effect on the studies that are published about the efficacy of seasonal flu vaccination, which is also widely cited by anti-vaccinationists, including Barbara Loe Fisher herself. Don’t get me wrong; publishing those studies does not make Jefferson in any way “anti-vaccine,” nor is the possibility of one’s work being abused by cranks for their own ends a reason not to publish. However, Jefferson seems quite naive if he thought that he was invited for any reason other than using him to apply a patina of scientific respectability to the rank pseudoscience much of the rest of the NVIC conference promotes.
Fortunately, I have learned from a private source whom I trust that, in response to e-mails to him pointing out the true nature of the NVIC, Dr. Jefferson has pulled out of the conference and will not be speaking there. I wish I could say more to back up my assertion right now. For the moment you’ll just have to take my word for it because Jefferson’s name is still prominently listed on the NVIC website, but I assure you that he will not be speaking there. The reasons he gave mainly emphasized his displeasure at being on the same stage with Wakefield, whose abuse of his research to support anti-vaccine nonsense greatly annoyed Jefferson; and his desire not to be used and manipulated by the NVIC. Good for him! Beter late than never! But that still leaves three others who are scheduled to speak.
These include Peter Doshi, Visiting Researcher, Faculty of Medicine, University of Tokyo and a doctoral candidate at Massachusetts Institute of Technoloy (MIT) in History, Anthropology, and Science, Technology and Society, who is known for skepticism about whether the bird flu would become pandemic and has written commentaries stating as such, one of which revere has roundly criticized. He is also known for claiming that the CDC intentionally inflates the death figures from influenza and that pandemic flu is no worse than seasonal flu. So it’s not surprising that he was invited. Moreover, I know that the same source from whom I learned of Jefferson’s withdrawal from the NVIC conference has also contacted Doshi and been rebuffed with a lame “my speaking there does not imply endorsement” gambit. Doshi will speak at the NVIC conference.
Next up is Diane M. Harper, MD, MPH. She’s an Associate Professor in the Department of Obstetrics/Gynecology and Community & Family Medicine at Dartmouth Medical School in Hanover, New Hampshire; serves in the Women’s and Gender Studies Department of Dartmouth College; and is the Director of the Gynecologic Cancer Prevention Research Group at the Norris Cotton Cancer Center in Lebanon, New Hampshire. She directs the referral colposcopy/high resolution anoscopy clinics at DHMC. Dr. Harper is know for her views on the HPV vaccine (Gardasil in the U.S.). It’s not that she thinks the vaccine is not effective or that it’s unsafe. She doesn’t, although I do think she overstates the risk of Guillan-Barré syndrome from the vaccine. Rather, Dr. Harper questions whether Gardasil should be incorporated into the adolescent vaccination schedule and promoted the way it’s being promoted, given that the regular Pap smears most women in industrialized countries undergo produce similar benefits against cervical cancer:
However, there is also another very important part to the cervical cancer prevention story, Dr. Harper said, and that is regular Pap tests. Even women who are vaccinated need to have regular Pap testing, as otherwise they are still at risk of developing cervical cancer. And women who decide not to have the vaccine can still protect themselves by undergoing Pap testing.
Dr. Harper feels this message has not been made clear to the general public and that it has been overshadowed by what she considers to be aggressive and inappropriate promotion of Gardasil. As a gynecologist dealing with the general population, her advice on the HPV vaccine is that “if you are at all concerned, then don’t have the vaccine — have regular Pap smears and you will be equally protected from cervical cancer.”
In other words, she thinks it’s a potentially useful vaccine for countries where women don’t get regular Pap smears but its benefits are not nearly as impressive in industrialized countries. I actually happen to agree with her (mostly) in that I’m not yet convinced that Gardasil’s benefits are worth the expense or that the risk-benefit ratio of the vaccine is so overwhelmingly in favor of the vaccine as MMR and other common childhood vaccines. However, unfortunately, at times Dr. Harper’s rhetoric has not been quite so measured, particularly when she has blasted Merck and referred to the vaccine as a “great big public health experiment.”
It’s fairly easy to figure out why Dr. Harper was attractive as a speaker for the NVIC Conference. I don’t think she’s anti-vaccine; she’s just skeptical of the HPV vaccine and its overpromotion, skepticism I share to some extent. However, I also don’t think that Dr. Harper has clue one about what the NVIC is and represents or that she is being brought in to bolster the NVIC’s anti-vaccine viewpoint. At least, I hope she doesn’t. Perhaps I will e-mail her a link to this post after it goes live and ask her, pointing out to her that she is being used.
Finally, there’s George Annas, JD, MPH. I don’t know much about him other than that he’s the Chair and Edward R. Utley Professor of the Department of Health Law, Bioethics & Human Rights at Boston University and that he is a highly respected lawyer and bioethicist. Doing a Pubmed search on his name only revealed five articles relevant to vaccines, the most recent of which was published in 2003 and was about bioterrorism and public health. He’s also written numerous books on bioethics, patients’ rights, and human experimentation, three of which have been listed on an anti-vaccine website as being recommended, although almost certainly not because Professor Annas is in any way anti-vaccine. My guess, given the title of his talk (“False Choices & Worst Case Scenarios: How Taking Informed Consent Seriously Can Improve Public Health”), is that the NVIC roped him in by asking him to talk about informed consent and vaccines (or just the issue of informed consent), without revealing the sheer amount of pseudoscience that will be presented in the very same conference. I suspect he will be quite disturbed at the talks he will see before and after him. Or maybe not. The day’s schedule around his talk appears carefully constructed to keep Dr. Harper, Peter Doshi, and Professor Annas carefully insulated from the likes of Wakefield, Ayoub, et al. If Professor Annas just flies in for one day to give his talk, he might not ever even realize the pseudoscience being promoted during the rest of the NVIC conference.
CRANKS AND THE APPEARANCE OF SCIENTIFIC RESPECTABILITY
If there’s one thing that cranks in general, and the antivaccine movement in paritcular, crave, it’s to be taken seriously by the scientific community. Moreover, although such cranks often show a marked hostility to science in that they ignore or attack scientific evidence that contradicts their beliefs (as well as the scientists who generate that data); rely on anecdotal evidence instead of epidemiology or randomized clinical trials; and engage in extreme conspiracy mongering about scientists and physicians who advocate science-based medicine. Yet, on the other hand, they desperately crave the appearance of science. To achieve this appearance, they adopt the conventions of science, including “scientific” studies that are not; crank journals to report the results of these “studies”; and “scientific” conferences like the NVIC Conference. When they can, they especially like to hold their conferences in a way that implies endorsement or support by a real scientific organization or a university, as Autism One has done with its Autism One Canada conference.
They also realize that it’s not enough just to have the same old cranks, like Wakefield, Ayoub, Rollens, et al. speaking at their conferences. They need to co-opt real scientists and real academics, like Drs. Jefferson and Harper and Professor Annas. My guess is that they were all persuaded to speak through appeals to individual rights versus public health (Prof. Annas) or controversies over the HPV and flu vaccines (Drs. Harper and Jefferson), with the invitations phrased in such a manner as to camouflage the true nature of the conference. That’s one thing that the NVIC is very good at. Alternatively, groups such as Autism One hide their real agenda behind terms like “biomedical interventions” and “complementary and alternative medicine” (CAM), coming across as just another CAM conference and even getting grants from reputable foundations like the SickKids Foundation and being allowed to hold their conference at a University of Toronto building.
Most physicians, academic or otherwise, or scientists doing research on topics like vaccines are not like me. They’re not like the bloggers here at SBM or even many of the readers in that the vast majority of them do not pay much attention to cranks like those in the anti-vaccine movement. It’s very similar to the situation with evolutionary biologists, the vast majority of whom also don’t pay much attention to attempts by creationists to deny and subvert the theory of evolution or to prevent its teaching in the public schools. They may be aware of the existence of such cranks but all too often consider them inconsequential, wondering how on earth anyone could take them seriously. So it is with the anti-vaccine movement. Morever, with the infiltration of pseudoscience into medical academia (what Dr. R. W. Donnell has termed “quackademic medicine,”) fueled by big money interests like the Bravewell Collaborative, “on the ground” it is becoming increasingly difficult for the average academic to distinguish pseudoscience from science, given the promotion of the latter by so many large academic institutions. Even when they do realize the difference, all too often academics adopt a “shruggie” attitude towards pseudoscience.
So what to do? Certainly, education is key. I highly doubt that either Professor Annas, Drs. Harper or Jefferson, or even Peter Yoshi knew when they accepted that the NVIC is an anti-vaccine organization or that they were invited that they would be used to provide the illusion of scientific or academic legitimacy to the conference. There are many venues in which to discuss legitimate scientific and policy controversies regarding vaccines, but the NVIC conference is certainly not one of them. It is a conference devoted to the manufactroversy that vaccines cause autism and all sorts of other neurological conditions, that vaccines don’t work, and that they do more harm than good, all cleverly couched as appeals to ethics, vaccine safety, and individual freedom. That’s why it is incumbent upon academics such as these, who express viewpoints that are easily co-opted by cranks, to be very careful about whom they endorse or where they speak, lest they taint their legitimate and useful criticisms of current science and public health policy with the stench of crankery. If anything, they should avoid being connected with such organizations because such connections make it easier for their opponents to dismiss them as cranks. Also, at the very least, they should avoid them because of the potential harm to their academic reputations. But, above, all they should avoid contact with such organizations because their presence at such events provides the illusion of legitimacy and actual scientific controversy that cranks crave. That is why I hope that Dr. Harper and Professor Annan–and, yes, even Peter Yoshi–will reconsider their previous acceptance of an offer to speak at the NVIC conference. At the very least, if they still end up speaking there, I hope they will be very careful not to give the NVIC any sound bites that they can use as “proof” that respected academics support the NVIC’s pseudoscience.
I’ll even take the risk that, should any of these academics actually back out of speaking at the NVIC conference, it will be trumpeted as evidence that “they” got to them and wouldn’t let them speak The Truth.

   How do scientists become cranks and doctors quacks?
David Gorski
Link
As a physician and scientists who’s dedicated his life to the application of science to the development of better medical treatments, I’ve often wondered how formerly admired scientists and physicians fall into pseudoscience or even generate into out-and-out cranks. Examples are numerous and depressing to contemplate. For example, there’s Linus Pauling, a highly respected chemist and Nobel Laureate, who in his later years became convinced that high dose vitamin C could cure cancer. Indeed, Pauling’s belief that high dose vitamin C could cure the common cold and cancer fueled the development of a whole new form of quackery known as “orthomolecular medicine,” whose entire philosophy seems to be based on the concept that if some vitamins are good more must be better. In essence, “orthomolecular medicine” is a parody of nutritional science; indeed, its advocates take credit for how some strains of “complementary and alternative medicine” (CAM) so frequently advocate the ingestion of huge amounts of dietary “supplements.” I could even go farther and say that orthomolecular medicine is clearly a major part of the “intellectual” (and I do use that term loosely) underpinning of the various biomedical treatments for autism that Jenny McCarthy and Generation Rescue advcoate.
There are other examples as well, all just as depressing to contemplate. For example, consider Peter Duesberg, a brilliant virologist who in the 1980s was widely believed to be on track for a Nobel Prize; that is, until he became fixated on the idea that HIV does not cause AIDS. True, lately he’s been trying to resurrect his scientific reputation with his interesting and possibly even promising chromosomal aneuploidy hypothesis of cancer, but, alas, true to form he’s been doing it by acting like a crank. Specifically, he sees his hypothesis as The One True Cause of Cancer and disparages conventional thinking as having been so very, very wrong all these years (with his being, of course, so very, very brilliant that he saw what no one else could see). Then there are people like Dr. Lorraine Day, who was a respected academic orthopedic surgeon in the 1980s. In the late 1980s, she started to flirt with AIDS pseudoscience through a scare campaign about catching AIDS from aerosolized blood. Of course, given the mystery and fear over HIV in the early years of the epidemic, such a fear, although overblown, was not so far out of the mainstream as to be worthy of the appellation crank. However, after being diagnosed with breast cancer, unfortunately Dr. Day rapidly degenerated into a purveyor of rank pseudoscience, as well as a New World Order conspiracy theorist, religious loon, and Holocaust denier. And let’s not forget Mark Geier, who, although not a distinguished scientist, did, before his conversion to antivaccinationism, apparently do a real fellowship at the NIH and appeared to be on track to a respectable, maybe even impressive, career as an academic physician. Now he’s doing “research” in his basement, injecting autistic children with a powerful anti-sex hormone drug and abusing epidemiology. There are innumerable other examples.
About a year ago, I saw a good discussion by a blogger by the ‘nym of ERV discussing how scientists can become cranks, and I’ve been meaning to expand upon it for a long time now. Since there didn’t appear to be anything happening this weekend that so inspired me to post, and Steve Novella already discussed recent articles in the AP about CAM and how, after an investment of over $2.5 billion, the National Center for Complementary and Alternative Medicine has failed to validate even a single CAM modality and, worse, how all of the negative studies of CAM coming out of NCCAM have failed to lead to the abandonment of even on CAM modality. In any case, as gratified as I am to see the mainstream media to be taking notice and starting to say the same sorts of things we’ve been saying on this blog since its inception, for instance, my discussion of the waste of taxpayer money on studies of homeopathy, among other magic. And, of course, Tim Kreider wrote an excellent takedown of Sharon Begley’s apparent belief that publication pressures and an insistence on basic science are keeping cures from sick babies, the other topic that caught my fancy.
Because her interest is HIV research, ERV was particularly interested in how a brilliant man like Peter Duesberg could have fallen so far into pseudoscience. However, because she is a basic scientist, her view lacks a bit when it comes to why physicians become cranks or quacks. Consequently, I thought I’d expand a bit on this topic and bring a medical perspective to the question. Because I am both a surgeon and a scientist, I routinely straddle both worlds (sometimes not being taken seriously in either). I’m not sure that this gives me any special insight, but it does give me a different perspective than ERV.
First, ERV shows great insight in pointing out that scientists are wrong all the time. Indeed, science can almost be defined as a system or a method of self-correction that brings us closer to an understanding of how nature works. An absolutely essential part of science, therefore, is that we scientists must test our hypotheses and try to falsify them. When we attempt to do so, we find, in general, one of three results:
     1.  The hypothesis is not falsified.
     2.  The hypothesis is falsified.
     3.  The results are not sufficiently clear to falsify or support the hypothesis.
When a hypothesis is not falsified, generally scientists will either try to find new ways of falsifying it until they are satisfied that it can withstand all reasonable challenges. Alternatively, they will build on it and refine it based on their experiments, after which they try to falsify the new iterations of the hypothesis until they succeed. If the initial hypothesis is falsified, scientists generally will move on to a new hypothesis. True, they may not do so quickly or easily; after all, scientists are human too and just as prone to becoming emotionally attached to their pet ideas and favorite hypotheses, but move on they generally do–eventually. Admittedly, in the case of medicine it sometimes takes the rise of a new generation of physicians and the retirement of the old before older concepts of disease completely disappear from medical practice, but disappear they eventually do. (The wag in me can’t resist pointing out one exception to this rule: aalternative medicine, where prescientific ideas dating back hundreds, if not thousands, of years have been resurrected into seeming respectability under the mantle of “complementary and alternative medicine” or “integrative medicine,” which “integrates” these prescientific notions with scientific medicine. But, as I so often do, I digress.)
Of course, result #3 is the most common result; the answer is usually not immediately clear. Indeed, this uncertainty may persist for years, if not decades, before some scientific questions are resolved. this is what can happen with true scientific controversies (as opposed to manufactroversies, like whether vaccines or mercury in vaccines can cause autism). Once again because scientists are human, they can be quite rancorous, on rare occasions even escalating to the point of scientists yelling “bullshit!” at each other at seminars and scientific meetings. (Such meetings can actually be kind of fun, at least if you’re not at the receiving end. Actually, they can sometimes be fun even if you are at the receiving end.) Over time, however, evidence will accumulate, and experimental results will start pointing towards an answer. Sometimes a dramatic result, a stroke of genius, like Einstein’s Theory of Relativity or Charles Darwin’s Theory of Evolution by Natural Selection, will appear, seemingly like a bolt out of the blue and change everything. (I say “seemingly” because even such results do not appear in a vaccuum; there are almost always multiple scientists working along similar lines and it’s scientists like Einstein or Darwin who either publish first or somehow become the person given the most credit.) However it happens, though, once a hypothesis is roundly falsified by multiple lines of evidence (or, as in the case of the autism/vaccine link, numerous studies fail to find support for a link), scientists will indeed, even if very reluctantly, admit that the hypothesis was incorrect, form a consensus, and then move on to test other hypotheses. Or, as ERV put it:
Now here is where we get to the kook part– Say we finally determine that MMTV does not cause any kind of breast cancer in humans. All the normal scientists on the ‘MMTV causes cancer’ side will say ‘Aw. Man. Oh well, my bad.’ and keep doing science. Admitting youre wrong in science is not a big deal because scientists are wrong all the time!
Perhaps the best illustration of this attitude among scientists was a tale told by Richard Dawkins in The Root of All Evil? about an elderly and esteemed scientist who had held to a certain hypothesis for many years. One day a visiting professor from America came to give a talk and presented evidence that conclusively refuted this professor’s favorite hypothesis. As the lecture concluded, all eyes were on the esteemed senior scientist whose favorite hypothesis had just been roundly shown to be incorrect. According to Dawkins, the old professor strode to the podium, shook the speaker’s hand, thanked him profusely, and said, “I have been wrong these fifteen years.” In response, the audience applauded uproariously. Whether this story is apocryphal or not or whether it’s grown with Dawkins’ retelling it over time, it is nonetheless the ideal towards which science strives. Scientists are supposed to be willing to give up even their most cherished hypotheses if that’s what evidence and experimental results demand. Of course, the difficulty in doing so tends to be proportional to both the length of time the hypothesis has been cherished and the intensity of attachment. Now, here’s where “scientist” transitions to “kook”:
Where one transitions from ‘scientist’ to ‘kook’ is the inability to say ‘I was wrong’ and move forward. In the 1980s, Peter Duesberg could have been right! No one knew what was causing this scary-ass world-wide epidemic. Yeah, it might have been HIV-1, but in the US it could have been a side-effect of some recreational drug, only surfacing when use was wide-spread and intense enough. Duesberg could have been right!
But as time went on, evidence stacked up on the side of the ‘HIV-1–>AIDS’ people, and nothing supported the ‘drugs–>AIDS’ people. If Duesberg were a scientist, he could say ‘Ek. Im wrong. Good on you all.’ and continued his career.
Indeed, assuming that Duesberg had done this before he had burned too many bridges with his colleagues by so harshly criticizing and insulting those whom he perceived as part of what he dismissively called the “orthodoxy,” it probably would indeed have been “no harm no foul,” and Duesberg would have resurrected his career and remained a happily well-funded scientist doing important research, rather than a tenured crank working in crappy basement laboratory on a shoestring budget. Heck, even after Duesberg had burned a lot of those bridges, he probably could have rebuilt them with some really good science leavened with just a little diplomacy directed at those whom he had previously attacked. But he didn’t, because, like all cranks, he had become so enamored of his hypothesis that he was unable to let go of it even after the contradictory evidence had become overwhelming. He had become convinced that he was right and the world of HIV scientists were (and continue to be) oh so very wrong.
The other point that is worth emphasizing is that being a contrarian is not in and of itself particularly impressive because scientists are so often wrong. There are far more hypotheses that are falsified than hypotheses that stand up to experimental and observational scrutiny. Indeed, I find “contrarian” scientists who won’t support their doubts of the established consensus with good science (and sometimes not even good logical arguments) of their own to be particularly annoying, like a two-year old who says “No!” to everything. Being “contrarian” is only productive if the contrarian scientist can produce actual evidence using sound experimental and observational methodology suggesting that the consensus is seriously wrong. That’s one reason why “intelligent design” creationists (or, as I’ve increasingly started calling them, evolution denialists) are not taken seriously and should not be taken seriously. They point out what they see as “shortcomings” in evolutionary theory, some valid but most based on gross misunderstandings of what evolutionary biology actually says, and do no research. Indeed, they don’t even try to do any research that might suggest alternatives. The same is true of cranks of all stripes, including “alt-med” cranks, HIV/AIDS denialists, and many other varieties.
So basically the key ingredients of a scientific crank are an inordinate attraction to an idea or hypothesis to the point that he won’t abandon it in the face of overwhelming evidence coupled with the arrogance necessary to believe that he is correct and the rest of the scientific community is not. This is probably true no matter what sort of science is being abused. However, when it’s medical science that leads to outright quackery, there are other issues that come into play.
The first thing that one needs to take into account when considering the evolution of a medical crank is that most physicians are not scientists. As much as it pains me to admit it, sadly it’s true. Indeed, I have lamented time and time again how little training in the scientific method most medical students and residents receive. This makes all too many physicians very susceptible to pseudoscience because they don’t have a good grasp of what good scientific methodology is, and I’ve provided ample examples of this in this blog, perhaps most prominent of which is Dr. Jay Gordon, who rejects the science and epidemiology that have failed to find a detectable correlation between either mercury in the thimerosal preservative that used to be in vaccines or vaccines themselves and autism in favor of his “personal clinical experience,” to the point that he frequently shows up on TV as a “vaccine skeptic” and writes articles arguing against holding anti-vaccine parents legally responsible if their child infects another. In other words, he favors anecdotes over science and, no matter how many times I’ve tried to convince him how easily we as humans can be deceived into confusing correlation with causation or into thinking that a useless treatment works, he cannot seem to accept that he can be deceived as easily as you or I.
Indeed, from my perspective the sine qua non of crank is a certain arrogance. Among medical cranks in particular, this arrogance manifests itself in the failure to acknowledge just how easily all humans, including them, can confuse correlation with causation, engage in selective memory such as confirmation bias, and are fooled by anecdotes, personal experience, and regression to the mean. I can understand how this can happen. Even among people steeped in the scientific method, it is sometimes hard not to fall prey to these shortcomings in human cognition. It is this tendency that will lead them to liken themselves to Galileo, persecuted scientists who will someday be vindicated. They will invoke Ignaz Semmelweis, whose evidence showing how handwashing could greatly reduce the rates of pueperal fever in the obstetrics wards was initially rejected by much of the medical establishment, and the more recent example of Barry Marshall and Robin Warren, who discovered that most duodenal ulcers were caused not by acid but by an organism called H. pylori.
Of course, the case of Semmelweis is more complex than usually proffered by cranks in that he was hesitant to publish and present his results and tended to antagonize his critics, coupled to the lack of an explanatory mechanism for his observations, given that his discovery was made before Pasteur’s germ theory of disease, leaving only the “contagion” concept as a possible explanation. The case of Marshall and Warren is also not a good example in that, although there was initial skepticism of their results, in actuality Marshall and Warren essentially won the day within a decade, by which time the standard of care for treating duodenal ulcers had evolved to using antibiotics. Since then, ulcer surgery has declined to the point where fewer and fewer surgeons even know how to do it anymore except in the case of emergencies (such as perforation or bleeding), so rarely is it necessary. That’s actually an incredibly fast about-face for medicine, given how long it typically takes to bring an idea from bench to bedside. In any case, even if we view the cases of Galileo, Semmelweis, and Marshall and Warren were exactly as described, examples of iconoclast scientists triumphing over the orthodoxy, that would not support the quacks who like to invoke them. As Michael Shermer once said, “Heresy does not equal correctness.” Shermer also correctly put it in his book Why People Believe Weird Things (a book I highly recommend to anyone interested in improving his or her critical thinking skills):
For every Galileo shown the instruments of torture for advocating scientific truth, there are a thousand (or ten thousand) unknowns whose ‘truths’ never pass scientific muster with other scientists. The scientific community cannot be expected to test every fanstastic claim that comes along, especially when so many are logically inconsistent.
Or, as Robert Park put it:
Alas, to wear the mantle of Galileo it is not enough that you be persecuted by an unkind establishment; you must also be right.
Cranks want to claim the mantle of Galileo without evidence to show that they are correct.
The second thing that needs to be considered is that there is a component of being a physician that doesn’t exist for basic scientists that provides added impetus to the transition from scientist/physician to kook, and that component is direct interactions with patients. This component is a powerful contributor to physicians becoming what Prometheus likes to call them, “brave maverick doctors.”
Consider this: Despite what many who do not like “conventional” physicians say, the vast majority of physicians go into medicine because they want to help people. They honestly want to help their patients, and not being able to help their patients causes them intense feelings of inadequacy and disappointment. In other words, there is a strong incentive, both positive (to get that fantastic feeling we get when we realize that we’ve really helped a patient) and negative (to avoid that feeling of inadequacy and frustration that we get when we cannot help a patient), to find treatments that the physician perceives to help patients. There’s also the ego gratification that comes from patients telling one that he’s great and has helped him enormously. The problem with this desire, however noble, is that it makes physicians very susceptible to the siren call of pseudoscience in the form of quackery. Here’s why. “From the ground,” a single physician looking at a group of his or her own patients being treated with a therapeutic modality will almost always “see” that the therapy “works.” The reasons, of course, include the placebo effect and regression to the mean. Without a controlled clinical trial, the placebo effect will almost always bestow upon almost any therapeutic modality at least the illusion of therapeutic efficacy, particularly at the single practitioner level. Thus, it is not surprising that homeopathy, reiki, acupuncture, and all manner of modalities that clinical trials show to be no more effective than placebo can easily appear to be effective when a single practitioner does what is in essence an uncontrolled, single-arm observational trial by treating patients with these modalities. Add to these human tendencies a significant financial incentive, and one can see how the slide into quackery can start out as an exercise in idealism, with “start out” being the operative phrase.
For example, for all the venom I’ve dripped on Dr. Mark Geier (venom he richly deserves for using powerful antiandrogenic drugs on autistic children), I have little doubt that he probably honest believes that he is helping autistic children with his “Lupron protocol,” as autism is a condition of developmental delay, not stasis. Many autistic children improve simply with growth and development, and without a control group and controlled, blinded randomization it’s impossible to tell whether a given intervention leads to an improvement in symptoms over and above what would have occurred anyway. Unfortunately, this belief has led them into some incredibly unethical activities in the service of their belief that they are somehow “helping” autistic children and with the claim that they “know” it works. Indeed, it is rather interesting to read Dr. Geier’s statement in defense of Clifford Shoemaker, a lawyer who has made a cottage industry of bringing complaints to the Vaccine Court, in which he complains of Kathleen Seidel’s activities against his pseudoscience as though he’s a poor, persecuted crusader for good. Of course, poor crusaders for good rarely charge the exorbitant fees that Dr. Geier charges or skirt human subjects research ethics the way he does, but in his world-view he is a lone iconoclast fighting the good fight against an unyielding orthodoxy.
This is where physician cranks (i.e., quacks) then become just like cranks of all scientific stripes, the sole exception being that their crankery endangers patients. They no longer try to falsify hypotheses or do trials to figure out if their remedies work. Why should they? They know to the very core of their being that they work! Instead, such physicians cherry pick studies that support their idea and, if they do any clinical studies or science at all, it is almost universally bad science. Over time, they come to believe that they are right “because I see the evidence in my patients,” as quacks like to say, a statement echoed in the statement of a nonphysician like Jenny McCarthy, who has been widely quoted as saying that her son Evan “is my science.” They forget utterly how easy it is to be fooled by a combination of the placebo effect, expectation, confirmation bias, and observation effect when one is a single physician treating patients. Whether it’s preexisting arrogance or a developed arrogance, these physicians will then often dismiss as “sheeple” the physicians who practice science- and evidence-based medicine as lacking the vision that they have, all the while making excuses for not doing rigorous clinical studies that would confirm or disprove the efficacy of their remedies over and above a placebo. Some, like Dr. Rashid Buttar, will charge enormous fees for their services and think it justified to the point that when they are questioned about the evidence supporting their therapies they will dance around the question and when they are finally brought in front of medical boards for their activities they will behave as an aggrieved, persecuted party, sometimes even going so far as Dr. Buttar as referring to the medical board as a “rabid dog.”
I tend to consider physician-cranks to be almost a special case of scientific cranks in general. In no other field of which I’m aware can the combination of an attraction to a hypothesis and arrogance combine with an honest desire to help patients in a manner that is so toxic to both science and patients. For most scientists, the forces that seduce them into pseudoscience are largely a combination of intellect and arrogance. For physicians, a genuine concern for patients and the placebo effect enter into the equation to form a perfect storm that can tempt them into even the most indefensible pseudoscience.

   Help – My Doctor is a Crank!
Steven Novella
Link
I often receive e-mail from SBM readers (or SGU listeners) who have had the experience of their doctor, nurse, dentist, physical therapist, or other health care provider recommending to them a treatment option that seems dubious, if not outright pseudoscientific. They want advice on what to do. There are common themes to the e-mails – the writer often feels very uncomfortable in the situation. They do not feel comfortable confronting their provider directly, yet they do not want to acquiesce to the advice either. They are also often asking my opinion about the advice – is it really as wacky as it seems. This uncertainty saps them of their resolve, leaving them feeling a bit helpless.
Here is one such e-mail:
Ten days ago, my wife and I welcomed our first child into the world. She was born a couple weeks early, which left her mouth a bit too small and week to breastfeed effectively. To prevent her from losing too much weight, we were referred to a lactation consultant (who works out of the pediatrics department at the hospital where our daughter was born). This consultant (who is also an RN) suggested a regimen of supplementing nursing with pumped breast milk.
This was working great until my wife’s milk production dropped the day before our follow-up appointment. When we asked what to do about this, the nurse recommended that my wife take fenugreek, an herbal supplement. I was a bit skeptical of this advice, so I asked what it was about fenugreek that helped with milk production. The lactation nurse’s answer was vague — she said things like, Herbs can be helpful for lots of health issues, and, a lot of women I see seem to think it helps (oh, the logical fallacies). When we pushed her on this a little more, she handed us a flyer, printed by the hospital about fenugreek. The flyer seemed to support the use of the supplement, but mentioned that there was no scientific research demonstrating that fenugreek increases milk supply. When we asked why it hadn’t been researched, the nurse responded that there wasn’t a lot of money in lactation and that scientists generally aren’t interested in the kind of things she does (basically, that she was doing the good work that cold-hearted scientists refused to do).
She also mentioned that there was a prescription drug that boosts milk supply, but it carries with it the risk of a pretty serious side effect (depression), so she prefers her patients to try the supplement first. I asked how we could be sure the supplement didn’t also carry the risk of negative side effects if it hadn’t been studied, she simply said that it worked out fine for the patients she’s seen.
My wife and I decided to let it go for the time being and decide later whether or not to go the supplement route (it’s hard to argue with a nurse when you’re under-slept, one of you is topless, and the baby is screaming for food). Fortunately, her milk supply picked up a bit on its own, so we didn’t have to resort to using an untested therapy (yet).
First, let’s address the recommendation – fenugreek to increase breast milk production. Substances that increase milk production are called galactogogues. There are a number of drugs that have been shown to increase milk production, although the levels of evidence varies. For many there are only case series. Oxytocin is supported by double-blind placebo controlled trials, and is both safe and effective. Other commonly used drugs, like metoclopramide, are supported by unblinded case series only.
Fenugreek is a common spice and medicinal herb used in India. Among its putative effects is increased lactation, however, there are no clinical trials to support its use. Recommendations are based upon anecdotes only. This 2010 review article reports:
The side effects most commonly reported are a maple-like smell of the urine, breast milk, and perspiration, diarrhea, as well as the worsening of symptoms in individuals with asthma or hypoglycemia. The potential for transfer to milk or side effects in the infant are unknown. As is the case for most herbal products, the dose necessary to obtain a galactogogic effect has not been defined. Only one study on the effect of fenugreek on lactation has been reported. Swafford asked 10 mothers to maintain a diary of the quantity of milk produced with a pump for a period of two weeks. In the first week, baseline milk production was evaluated; in the second week, mothers took fenugreek, 3 cups, three times daily. In the first week average quantities were 207 ml/day, whereas, milk production in the second week averaged 464 ml/day (p = 0.004). Unfortunately, the report excluded any information regarding the characteristics of the mothers enrolled in the study or the postpartum period during which the study was done. Nevertheless, the daily milk quantities reported during treatment do not seem to be particularly high (reference: Swafford S, Berens P. Effect of fenugreek on breast milk production. Abstract 5th International Meeting of the Academy of Breastfeeding Medicine September 11-13, 2000, Tucson, Ariz Academy of Breastfeeding Medicine News and Views 2000;6(3).)
So – there are side effects, even potentially serious side effects like exacerbating asthma and hypoglycemia. The dosing is unknown, and the one small efficacy study that was done was unimpressive.
There is nothing implausible about the claim that an herb can be an effective galactogogue – herbs can be drugs with pharmacological activity. What is disturbing about the nurse’s recommendation is the double standard. She seems to believe that herbs are inherently safer than drugs, which is a false dichotomy based on the naturalistic fallacy, but not logic or evidence. She essentially recommended an untested drug with uncertain dosing, side effect, or efficacy over drugs with better evidence for both safety and efficacy.
It should also be noted that the e-mailer’s mild production spontaneously increased after the consultation. If she had taken fenugreek, that would have been one more anecdotal report apparently supporting its efficacy.
But onto the real question of the e-mail – how to respond in this situation. My advice is to first not be intimidated. Feel free to express your concerns or uncertainty about the recommendations being made. I commend this e-mailer for asking for published evidence, but don’t take an evasive answer as adequate. Ask for published evidence or authoritative reviews. If they do not have any handy, they can certainly prepare this material for the next visit or simply e-mail it to you.
If you feel up to the task, you can research the question yourself and then ask your health care provider to comment on the material you find (but don’t overwhelm them with mounds of material).
Depending on how egregious the pseudoscientific advice was, you should also consider simply leaving that provider for someone with whom you feel more comfortable and confident. I do recommend, whether or not you stay with the provider, to give them feedback. Express your exact concerns about their advice. How they respond will also tell you a great deal about their approach and dedication to evidence-based practice.
In short, I think patients should feel empowered to push back against practitioners who stray from science and reason as a basis for health care recommendations and practice. Those pushing for anti-science in medicine are certainly vocal, and their agenda will advance if others simply do nothing. In fact – if a practitioner recommended an outright anti-scientific treatment (like homeopathy, say) I would go beyond giving them individual feedback. I would write to the head of their clinic, hospital, or department and express your concerns. You are a consumer as well as a patient, and administrators listen to their consumers.
The above situation is becoming distressingly common, and it is not easy to deal with. It is one more type of harm that is caused by the infiltration of anti-science into medicine. I had a similar experience myself. My wife and I were referred to a practitioner to address a problem with one of our children (I am being deliberately vague on details). At the first visit the practitioner made a comment (as an aside, not directly related to the visit) about vaccines and autism. I, of course, politely challenged her on that statement, and she backpeddled – I think just to end the confrontation. My wife thought that I was inappropriate to challenge her – she was concerned about my accepting the role as patient rather than physician.
At the next visit, however – the visit when the practitioner was giving us her actual recommendations, my wife had a change of heart. The practitioner’s recommendations were entirely limited to “natural” interventions like diet and herbs, and she shied away from any medications. Her recommendations were ideology-based, not evidence-based, and in the end the consultation was entirely useless. My concerns at the first visit – that perhaps this practitioner does not have a firm grasp on the concepts of science-based medicine, were entirely vindicated.
This is an important point – if a practitioner recommends something that is blatantly against the science and evidence, how can you have confidence in any of their recommendations? I am not talking about not being up on one bit of latest evidence – no one can know everything. A serious lapse, however, can reveal a systematic bias in a practitioner’s approach to evidence and practice.
Unfortunately, patients cannot take for granted that a licensed professional is necessarily science-based.

   Integrative Obfuscation
Steven Novella
Link
The marketing of so-called CAM or integrative medicine continues. These terms are just that – marketing. They are otherwise vacuous, even deceptive, and meant only to conceal the naked fact that most medical interventions that hide under the CAM/integrative umbrella lack plausibility or credible evidence that they actually work.
Medicine that works is simply “medicine” – everything else needs marketing.
Last week in the British Medicine Journal (BMJ) Hugh MacPherson, David Peters, and Catherine Zollman wrote a very telling editorial entitled Closing the evidence gap in integrative medicine, which Edzard Ernst has rightly characterized “a masterpiece in obfuscation.”
The essence of the editorial can be boiled down to this – proponents of integrative medicine are disappointed that scientific research has not validated their failed modalities. Therefore they want to weaken the rules of evidence so that they can get the results they desire.
But first, they begin their marketing with the branding of “integrative medicine”, taken from www.imconsortium.org:
“medicine that reaffirms the importance of the relationship between practitioner and patient, focuses on the whole person, is informed by evidence, and makes use of all appropriate therapeutic approaches, healthcare professionals and disciplines (conventional and complementary) to achieve optimal health and healing”
This is not integrative medicine, which is rather the attempt at mixing unproven, disproved, and often highly implausible methods into science-based medicine. But proponents can’t go around saying directly that they want to use treatments that probably don’t work, so they brand a nice sounding term with a collection of virtues that actually have nothing to do with what they are selling.
This is obfuscation because a definition such as this should indicate what distinguishes one category from another – what makes integrative medicine different from medicine. The practitioner-patient relationship has been central to medicine since Hippocrates. The “bio-psycho-social” (whole person) approach to medicine is also nothing new – that’s what I was taught in medical school before the term “integrative” was invented. “Informed by evidence?” – I hope so. In fact, that’s a bit weakly stated. And mainstream medicine already makes use of all “appropriate” therapies – the addition of the world “complementary” is just adding another vague term without any operational definition.
So what is the difference between “integrative” medicine and medicine? None that I can detect by that statement, except the suggestion (well-hidden) of using unproven treatments. The brilliance of the marketing, of course, is that by defining your made-up brand of medicine with these virtues, you suggest that your competition lacks them – without ever having to say it.
Let’s get to the meat of the editorial. They write:
Yet when it comes to deciding whether an intervention, and which type of intervention, might be helpful for a particular patient, a worrying gap exists between the perceived potential for using integrative approaches in areas of poorly met clinical need and the availability of supporting evidence derived from good research.
Throughout the editorial the unstated major premise is that the authors know that the “integrative” modalities they want to promote work. They simply need to figure out a way to support what they already know to be true with something that can be marketed as scientific evidence. Nowhere do they give a hint that they are concerned about using science to figure out IF a treatment works.
They also do not give the slightest consideration to the fact that the evidence gap may be the result of the fact that the treatments simply do not work.
In a response to the editorial, David Colquhoun correctly points out their “special pleading” – “If your treatments cannot pass the test, the test must be wrong.” The test they are complaining about is the randomized controlled trial.
They make the usual excuses for why rigorous scientific studies are not adequate to evaluate integrative methods – they are too complex, the studies are artificial and don’t apply to the real world, and they interfere with choice and the therapeutic relationship. This is flawed reasoning on multiple levels.
First, many “integrative” methods are quite amendable to the double-blind randomized trial. Anything that can be taken as a pill – such as herbs or homeopathy – can be compared to an identical-looking placebo. Procedures are more challenging, but researchers have figured out ways to compare acupuncture to sham and even placebo acupuncture, using non-penetrating needles in an opaque sheath.
But on a deeper level they miss the point of scientific research entirely. They complain that the therapeutic relationship is part of the treatment (as if that is different than regular medicine) and therefore it cannot be separated from the treatment itself in clinical trials. But the entire point of scientific studies is to isolate variables as much as possible so that we can come to conclusions about whether or not each variable has a net positive or negative health effect in certain situations and conditions. This is what makes medicine scientific an allows it to move forward – knowing what works and is safe and what isn’t. The art of medicine is then putting it all together in a positive therapeutic interaction.
James May, chairman of HealthWatch, nailed it when he wrote in response:
Effective medicine is best measured with RCTs. Caring is not. ‘Integrative medicine’ therefore risks both damaging how we measure effective medicines (RCTs), as well as reducing caring to measurables. A better term for this might be ‘disintegrative medicine’.
He is saying, as I just did, that the authors miss the point about the purpose of medical research.
In the place of the randomized controlled trial, the authors recommend so-called “pragmatic” studies, comparison research, and case studies. These all have one thing in common – they are not blinded. They therefore do not separate out the variable of the intervention in question from the therapeutic ritual and non-specific effects. These kinds of studies are useful for guiding the application of treatments that have already been shown to work. But they are useless in determining if a treatment works.
Conclusion
Over the last couple of decades there has been a persistent effort to promote unscientific medical modalities with increasingly sophisticated marketing – and to the horror of many, the marketing is working. Proponents have come up with an steady stream of euphemisms for medicine not supported by science – “alternative”, “complementary”, “integrative”, and “functional”. They have marketed them as “natural” and “holistic” and assured the public that if we just researched these methods they would be proven to work. They marketed their modalities to the public, to regulators, and to academia.
Now, after decades of research, after the NCCAM spent 2.5 billion dollars of taxpayer money – the research overwhelmingly shows that these implausible methods don’t work. (It should not have been surprising that treatments that had a very low prior probability turned out not to work – the improbable, it turns out, is improbable.) For years we were told that we just needed more research and better designed research. Acupuncture studies need better placebo-controlled acupuncture.
Now “integrative” proponents are switching gears (after the research they were asking for did not turn out as they hoped), increasingly arguing that the problem is not their improbable modalities but the research technology itself. They want to switch to the kind of studies that are not even designed to determine if a treatment works or not, and are almost guaranteed to produce the results they want – results they can use to confuse the public and politicians, and to lubricate their passage into the halls of academia.
To this skeptic, this is an old and familiar strategy. When science fails to validate your belief system, blame science.

Tribalism and Medical Ethics
Harriet Hall on January 14, 2014
Link
Science is intended to discover the “is”, not the “ought;” facts, not values. Science can’t tell us whether an action is moral; it can only provide evidence to help inform moral decisions. For instance, some people who believe abortion is immoral reject birth control methods that prevent implantation of a fertilized ovum on the grounds that it constitutes abortion; science can determine that a particular birth control method prevents fertilization rather than preventing implantation of a fertilized ovum. A new book, Moral Tribes: Emotion, Reason, and the Gap Between Us and Them, by Joshua Greene, provides some intriguing insights that are pertinent to medical ethics.
He thinks tribalism is the central tragedy of modern life. Evolution equipped us for cooperation within our own tribe but not for cooperation with other tribes. Cooperation with related individuals helps spread our own genes, but we are in competition with other tribes and cooperating with them might help spread their genes to the detriment of our own. It boils down to Us vs. Me and Us vs. Them. He uses the word “tribes” not in the original sense (Hutus vs. Tutsis), but to include Democrats vs. Republicans, Catholics vs. Protestants, CAM vs. science-based medicine, Arabs vs. Israelis, climate change activists vs. climate change deniers, and any other ideological or nationalistic group.
He says morality is not what generations of philosophers and theologians have assumed. It is not a set of freestanding abstract truths, but rather a manifestation of moral psychology.
    • Morality is a suite of psychological capacities designed by biological and cultural evolution to promote cooperation.
    • It is implemented through emotional moral intuitions, gut reactions that cause us to value the interests of (some) others.
    • Different human groups have different moral intuitions, resulting in conflict.
Opponents of abortion feel strongly that it is always “wrong” to destroy a fetus. Proponents of abortion feel just as strongly that it is always “wrong” to interfere with a woman’s control of her own body. Their positions are incompatible, and they are unable to agree on a policy that will satisfy both tribes. Greene thinks we can only make progress if we understand what morality is, how it got here, and how it’s implemented in our brains. He describes recent studies in experimental psychology and cognitive neuroscience that can help us understand.
Trolleyology
The science of “trolleyology” investigates the role of emotions in morality. The basic trolley scenario: a trolley is headed down the track towards 5 people, and will kill those people unless you throw a switch and send it down another track where it will only kill one person. Most people agree that it is better to throw the switch; but if the choice is to stop the trolley by pushing a person off a footbridge onto the track, most people wouldn’t do it. The math is the same: sacrificing one life to save five. But most people who are not psychopaths have a strong emotional reaction that makes them feel it is more immoral to actively push someone than to throw a switch. This is a gut reaction that doesn’t involve reasoning. But what if the person you push is a rapist and murderer? What if the one person killed by switching tracks is your child? Variations on the trolley theme are a good tool for investigating how our brains work.
We have dual-process brains
We have dual-process brains: we react automatically, but we can also think things through deliberately. The automatic mode allows us to react quickly and appropriately to danger (see lion, run); the manual mode allows us to weigh the consequences of our actions and defer temporary pleasures for long-term gain (stopping to eat berries vs. continuing to follow the trail of a mastodon that you can kill to feed the whole tribe.) A gizmo in our brains responds negatively to violent acts, regardless of whatever benefits they may produce. It also gives us a taste for punishing others when they act unfairly and makes us far more concerned about the plight of people who are present than about the plight of those who are far away. The gizmo has an important function; but we should recognize that while our moral intuitions are generally sensible, they not infallible.
Greene compares the brain to a camera with general-purpose automatic settings that can be over-ridden to customize the settings for specific purposes. Emotions serve as heuristics, mental shortcuts that we use to make decisions. They allow rapid, efficient action that is generally adaptive. Reason can free us from the tyranny of emotion, but it can’t produce the best decisions without some emotional input. Think Mr. Spock and Capt. Kirk. Emotions that are not tempered by reason can lead to irrational mistakes; both our perceptions and our sense of fairness are biased. Selfishness can be overcome by our automatic settings for the benefit of the tribe; moral inflexibility in the face of another morally inflexible tribe can only be overcome by shifting into manual mode.
Facts are not enough
We like to think that if people are given the facts and have critical thinking skills, they will recognize the truth about climate change, vaccination, etc. Not true. People who are more scientifically literate are more adept at defending their tribe’s beliefs, whether they are true or not. Ordinary people are hopeless slaves to tribal prejudice. People are willing to consult experts about most things, but expert consensus means nothing once a false belief has been culturally entrenched and has become a tribal badge of honor. Experts agree on climate change, but there’s widespread disagreement among the public. Egalitarian communitarians accept the scientists’ consensus of great risk and the need for collective action; hierarchical individualists trust their own tribe’s chosen high-status authorities and are wary of collective actions.
Many of our debates hinge on standard value premises of security vs. freedom. This becomes crystal clear in discussions of vaccines. Pro-vaccinationists value the security of eliminating preventable diseases for everyone; anti-vaccinationists value the freedom to make individual decisions and to put what they perceive to be their own personal interests above the interests of herd immunity.
Political divisions often boil down to individualism vs. collectivism. Should everyone pay the same flat income tax, or should the rich pay more than the poor? One side thinks the individual should reap the rewards of his own hard work and believes it is unfair to take goods from him and give to others who have not earned them. The other side thinks justice requires the more fortunate to share with the less fortunate. Both sides are more committed to their ways of life than to producing good results. Their values are not about “whatever works” but about deep moral truths.
Harms caused by action vs. harms of omission
We instinctively feel that harms caused by actions are worse than harms of omission. This can be debunked. Our brains can’t keep track of all the things we are NOT doing. The brain considers actions and omissions in fundamentally different ways, and actions are more accessible to our minds. The AMA distinguishes intentional euthanasia (forbidden) from providing drugs to relieve pain that also happen to end a patient’s life. Under the AMA Doctrine of Double Effect and the Doctrine of Doing and Allowing, terminally ill patients may not get what’s best and what they want for themselves. Our automatic gizmo creates opposition to physician-assisted suicide, mandatory vaccination, and policies surrounding organ donation and abortion.
We are working against 6 psychological tendencies
     1.  Human tribalism favors Us over Them
     2.  Tribes place different emphasis on rights of individuals vs. good of the group
     3.  Different groups follow different moral authorities, often religious
     4.  Biased fairness – group self-interest distorts sense of justice
     5.  Biased beliefs – from self-interest and social dynamics
     6.  We tend to underestimate the harm we cause others
Is utilitarianism the solution?
Greene thinks an enlightened utilitarianism is the answer: we should do whatever will produce the best overall consequences for all concerned. Caution is required, because it’s easy to fool ourselves into thinking that what we want serves the greater good. Some might argue that slavery could be an overall good if the advantages to the slave owners outweigh the harm to the slaves. It’s easy to show that belief to be false by simply asking whether they would agree to live half their life as slaves in order to reap the benefits of being a slave owner during the other half of their lives, especially if the order of events is randomly chosen and there is no guarantee of humane treatment for slaves. In principle you might be able to maximize happiness by oppressing some people, but in the real world oppression does not make the world a happier place. “In the real world, there is no fundamental tension between happiness and justice.”
Greene argues for Mill’s “higher pleasures.” Things like getting drunk or overeating are temporary pleasures that don’t benefit the individual in the long run; higher pleasures like the gratifications of learning are often those that also build durable and shareable resources. Our instincts serve us well to confront the moral temptations of everyday life (matters of Me vs. Us): they will stop us from hurting a neighbor by lying, stealing from him, or killing him. But when it’s a matter of Us vs. Them our instincts are not reliable and we should resort to explicit thinking about the greater good.
Three approaches to moral truth that fail:
     1.  Religion: relies on personal revelation not accessible to all; relies on different interpretations of God’s will.
     2.  Reason: founded on axioms. After centuries of trying, no one has found a serviceable set of moral axioms that are (a) self-evidently true and (b) can be used to derive conclusions that settle real-world moral disagreements. Reason can make our moral opinions more consistent, but it can’t tell us how to make trade-offs between competing values of different moral tribes.
     3.  Science: fails on the “is/ought” level. Evolution can explain how morality evolved to promote cooperation within groups for the sake of competition between groups. Our genes might be better spread by killing all the other tribe than by cooperating with them. The axiom “what’s right is what fulfills the purpose for which morality evolved” is not self-evidently true. Cooperation is not the ultimate moral good: it can result in misery for those who cooperate. The Borg is highly cooperative but isn’t much fun, and most of us would not choose to join. Cooperation is not an end in itself, but is valuable because of the happiness and relief from suffering that it brings.
Abortion
Greene quotes Barack Obama:
Democracy demands that the religiously motivated translate their concerns into universal, rather than religion-specific, values. It requires that their proposals be subject to argument, and amenable to reason. I may be opposed to abortion for religious reasons, but if I seek to pass a law banning the practice, I cannot simply point to the teachings of my church or [invoke] God’s will. I have to explain why abortion violates some principle that is accessible to people of all faiths, including those with no faith at all.
Abortion debate boils down to strong but complicated feelings that we can neither justify nor ignore. But we can ask questions about consequences. What impact do policies have on our lives? What happens if legal access to abortion is restricted?
If abortion is illegal, it removes an important safety net from millions of people, it tends to interfere with the enjoyment of sex, it causes people to seek illegal abortions at great expense or risk, it forces women to endure the discomforts and health risks of pregnancy against their will, it disrupts life plans, it tends to remove women from the work force, and it even costs lives (the death rate from pregnancy is higher than from therapeutic abortion). There’s no good argument against abortion except that it would grant life to people who wouldn’t otherwise exist. Those who are concerned about overpopulation would disagree that that was a good thing; and even for those who agree, this argument is TOO good: if it is valid, it should also apply to all forms of contraception and nonreproductive sex. Remember Monty Python’s “Every Sperm is Sacred.” In an unbiased assessment of the consequences, pro-choice wins, although it still hasn’t determined where to draw the line. If you go by fetal viability, the line changes as technology improves. Any line is arbitrary, but it could always be adjusted based on good evidence of harms vs. benefits. If our gut reaction is that early abortion is OK but 3rd trimester is not, we can look for a moral theory to match our feelings, but gut reactions were not designed to be organized, and they weren’t necessarily designed to serve truly moral ends.
Our reasons are confabulations
Science shows that most of the reasons we give are confabulations to rationalize preferences. People primed with words like ocean-moon were more likely to choose Tide detergent, but gave reasons like “Mom used it” or “I liked the box”. We constantly interpret our own behavior, building a plausible narrative about what we are doing and why. We rationalize to justify the emotionally-compelling moral dictates of our automatic settings.
Death penalty proponents and opponents cite evidence that it does or doesn’t reduce crime, but if they are proven wrong they fall back on “It’s still wrong on principle” or “It’s the moral right of an aggrieved society.” Heads I win, tails you lose.
Claims about what will promote the greater good (whether the death penalty will reduce crime, whether a policy will increase or decrease the total happiness of society) can be supported by evidence; claims about “rights” are oblivious to evidence.
People think they understand how things like zippers and toilets work but when asked to explain them, they realize that they don’t really know. In a brilliant set of experiments, people were asked about policy proposals like single-payer healthcare systems. Those who opposed a proposal were asked to explain the details of how it would work and were forced to realize they really didn’t fully understand it; this realization changed their attitudes and made them less opposed. People who were just asked for reasons for their opinion didn’t change. And we know that people whose reasons are challenged only become more convinced they are right. So forcing people to confront their ignorance of essential facts may be a way to make them more moderate. I wonder if it would help to ask anti-vaccine activists to explain how vaccines are developed and how they work?
Pragmatic utilitarianism as a solution to conflicts?
The first step to agreement is to concede that your opponents are not evil and hold their beliefs and values as strongly as you do yours. People are all basically alike. To me, I’m special; to them, they’re special too.
The next step is to set aside gut reactions and look objectively at consequences. We can capitalize on the values we share and seek a common currency there. The Golden Rule is generally accepted by everyone. Science also offers a common currency. Tribes may believe earthquakes are caused by the thrashing of giant catfish or the earth’s shivering when it is ill; such myths vary and contradict each other. But science is the same everywhere and we all ought to be able to agree that a scientific explanation based on plate tectonics is preferable to unsubstantiated myths.
How should we think about the foolish man who refused to buy health insurance? Is healthcare a right or a product? We can use manual-mode thinking to reach agreements with our “heads” rather than our “hearts” – to yield a moral philosophy that no one loves but that everyone “gets” – to acquire a second moral language that members of all tribes can speak. We can acknowledge that:
taking a bit of money from the haves hurts them very little, while providing resources and opportunities to the have-nots, when done wisely, goes a long way. That’s not socialism. That’s deep pragmatism.
Greene says:
Utilitarianism asks only that we push ourselves to be morally better, to care more than we do about people beyond our immediate circles. Utilitarianism doesn’t ask us to be morally perfect. It asks us to face up to our moral limitations and do as much as we humanly can to overcome them. Here science can help, showing us just how fickle and irrational our sense of duty can be.
This fits well with Michael Shermer’s hierarchy of morality in his book The Science of Good and Evil. He says we become progressively more moral as we expand our sphere of caring from self to immediate family to relatives to strangers to groups to nations to animals to the entire biosphere.
Conclusion
We have the capability to understand the psychology of our instinctive morality, overcome our automatic gut reactions when appropriate, and replace them with something better. We can aspire to overcome tribalism and become a global tribe that works towards the greatest possible happiness and welfare for all its members simply because it’s good. We can’t define good in any justifiable abstract philosophical way, but we can enhance the moral instincts evolution equipped us with to create a pragmatic utilitarianism that works for the benefit of everyone.
Greene’s ideas are applicable to controversies in medicine like abortion, euthanasia, circumcision, mandatory vaccination, universal healthcare coverage, and policies about CAM. They also apply to major global problems like poverty, violent conflict, terrorism, and global warming/environmental concerns and to domestic problems like taxes, capital punishment, gay rights, and many others. His book will make you think and will challenge you to recognize that you, too, have tribal biases. It’s well worth reading.

The World Health Organization promotes quackery yet again
The World Health Organization held the First WHO Traditional Medicine Global Summit this weekend. Unfortunately, its claims of being “evidence-based” aside, the conference followed the WHO’s usual pattern of serving as propaganda, not science. The summit was one-sided, organized by believers with the only speakers being believers, to promote a predetermined policy goal of promoting traditional medicine and justify “integrating” it with science-based medicine.
David Gorski on August 21, 2023
Link

The World Health Organization concluded its First WHO Traditional Medicine Global Summit on Friday. The conference was co-hosted by the Indian government and held in Gandhinagar, India, where the WHO had set up a Global Centre for Traditional Medicine, with help from $250 million from the Indian government. Unfortunately, as we have long lamented, the WHO has long had a penchant for promoting “traditional medicine,” particularly Traditional Chinese Medicine but also Ayurveda and others, as “evidence-based” and worthy of being “integrated” with science-based medicine, and this conference is just one more example of how far down that road the WHO has gone. To get an idea of how this meeting is being described and promoted by its stakeholders, I refer you to a statement by the Indian government released on August 17, the first day of the summit:
WHO’s first ever Global Summit on Traditional Medicine inaugurated as part of the G20 Health Minister’s Meeting in Gandhinagar, Gujarat.
By embracing ancient wisdom and modern science, we can collectively work towards achieving the health-related Sustainable Development Goals while fostering a ‘One Earth, One Family, One Future’ ethos: Dr. Mansukh Mandaviya.
“In modern times, the demand for natural and herbal-based pharmaceuticals and cosmetics underscores the enduring significance of traditional healing practices.”
“WHO’s Global Centre for Traditional Medicine headquartered in Jamnagar, Gujarat, accelerates advancements in Traditional Medicine globally.”
Traditional Medicine can play a huge role in honouring cultural diversity, empowering communities, and celebrating our shared heritage: Shri Sarbanand Sonowal.
I hope that the Gujarat Declaration, will integrate use of traditional medicines in national health systems, and help unlock the power of traditional medicine through science: Dr. Tedros Adhanom Ghebreyesus, WHO Director-General.
But what does the WHO itself say? Its statement announcing the summit listed the usual examples of medicines that derive from natural products used in some traditional medicine systems and touted how “evidence-based” the summit would be, before stating:
In response to this increased global interest and demand, WHO, with the support of the Government of India, established in March 2022 the WHO Global Centre for Traditional Medicine as a knowledge hub with a mission to catalyse ancient wisdom and modern science for the health and well-being of people and the planet. The WHO Traditional Medicine Centre scales up WHO’s existing capacity in traditional medicine and supplements the core WHO functions of governance, norms and country support carried out across the six regional Offices and Headquarters.
The Centre focuses on partnership, evidence, data, biodiversity and innovation to optimize the contribution of traditional medicine to global health, universal health coverage, and sustainable development, and is also guided by respect for local heritages, resources and rights.
Elsewhere, the WHO writes about its summit:
“Traditional medicine can play an important and catalytic role in achieving the goal of universal health coverage and meeting global health-related targets that were off-track even before the disruption caused by the COVID-19 pandemic,” said Dr Tedros Adhanom Ghebreyesus, WHO Director-General. “Bringing traditional medicine into the mainstream of health care — appropriately, effectively, and above all, safely based on the latest scientific evidence — can help bridge access gaps for millions of people around the world. It would be an important step toward people-centered and holistic approaches to health and well-being.”
Does this language sound familiar, particularly the parts about “ancient wisdom”? It is the language used by “integrative medicine” advocates to justify the “integration” of quackery with science-based medicine. Indeed, I immediately harkened back to an article I wrote for this blog nearly a decade ago about how Chairman Mao and his Communist government undertook a very systematic and intentional effort to retcon the history of Traditional Chinese Medicine. I’ll just drop a couple of quotes that I cited in that post for your edification:
“This One Medicine,” exulted the president of the Chinese Medical Association in 1952, “will possess a basis in modern natural sciences, will have absorbed the ancient and the new, the Chinese and the foreign, all medical achievements—and will be China’s New Medicine!”
Sound familiar? “This One Medicine” is about as simple a statement of what “integrative medicine” intends as I’ve ever encountered, and this one is 71 years old. Indeed, the WHO’s rationale that using “traditional medicine” to “bridge access gaps” is exactly the same as one major rationale that Chairman Mao used to promote traditional Chinese medicine. When he took power in the late 1940s, China did not have enough “Western”-trained doctors, leading to the idea of recruiting “traditional healers” (“barefoot doctors”), giving them some basic training in hygiene and more modern medical practices, and sending them to fill in the massive gaps in China’s healthcare system then.
I’ll cite another quote from Alan Levinovitz, whose writings in part inspired my 2014 article, in which he described how Chairman Mao understood that he needed to address the objections of proponents of scientific medicine:
His [Mao’s] solution was a two-pronged approach. First, inconsistent texts and idiosyncratic practices had to be standardized. Textbooks were written that portrayed Chinese medicine as a theoretical and practical whole, and they were taught in newly founded academies of so-called “traditional Chinese medicine,” a term that first appeared in English, not Chinese. Needless to say, the academies were anything but traditional, striving valiantly to “scientify” the teachings of classics that often contradicted one another and themselves. Terms such as “holism” (zhengtiguan) and “preventative care” (yufangxing) were used to provide the new system with appealing foundational principles, principles that are now standard fare in arguments about the benefits of alternative medicine.
I like to say that there is nothing new under the sun when it comes to antivaccine narratives, but it turns out that this principle also applies to narratives used to promote “ancient wisdom” as co-equal with science-based medicine. In essence, the WHO has been engaged in an effort to “scientify” (which I define as the effort to make practices based in prescientific mysticism and pseudoscience appear scientific) all of “traditional medicine,” as Mao tried (and, unfortunately, largely succeeded) to scientify TCM in the 1950s and as the Chinese government continues to do to this day. It’s an effort that has manifest itself in the WHO’s addition of “traditional medicine” diagnoses to the all-important ICD-11, the successor to the currently used ICD-10, a standardized system of alphanumeric codes for diagnoses maintained by the World Health Organization used throughout the world for billing, epidemiology, research, and cataloging causes of death. (Steve Novella described it as the WHO “endorsing quackery,” and he was correct.)
So let’s take a look at this conference. The entire proceedings can be found on video here.
The WHO’s promotion of traditional medicine vs. reality
Several days before the event, I noticed that the WHO had started promoting its summit on X, the social media platform formerly known as Twitter. I’m going to quote some of these Tweets because they reveal some very important things. Here’s the first part of the WHO’s promotion:
In 1969, Tu Youyou was tasked to search for antimalarial drugs. She turned to ancient Chinese #TraditionalMedicine books & discovered artemisinin. In 2015, Tu Youyou was awarded the Nobel Prize for her work on #malaria, which has saved millions of lives.https://t.co/TrcGF8rdeL pic.twitter.com/hqqn1D2idW
— World Health Organization (WHO) (@WHO) August 10, 2023
These are common rationales used by proponents of “traditional medicine,” namely that the existence of medicines derived from plants and other natural products, some of which were used in traditional medicine practices, validates the whole of traditional medicine. Of course, we’ve already discussed the example of Tu Youyou and why her winning the Nobel Prize for the discovery of artemisinin as an effective treatment for malaria does not somehow validate traditional Chinese medicine. Rather, it represented the triumph of natural products pharmacology, not herbalism or Traditional Chinese medicine. As Scott Gavura put it, alone “artemisinin isn’t effective as a treatment – it is eliminated too quickly from the body,” and added that what “finally turned artemisinin into a useful drug and brought this treatment to patients was in fact Big Pharma.”
The WHO continued:
For millions of people around the world #TraditionalMedicine is their first stop for health and well-being.
Which of these have you used?🖐️ Acupuncture🥣Ayurveda🌿Herbal medicine💊 Homeopathy🍃 Naturopathy💆‍♀️ Osteopathy🍵 Traditional Chinese medicine☀️ Unani medicine pic.twitter.com/VY9PUq7TMW
— World Health Organization (WHO) (@WHO) August 12, 2023
Aside from the repetition of Chairman Mao’s rationale for using TCM, I must say that it was definitely news to me that homeopathy, naturopathy, and osteopathy are in any way “traditional medicine.” After all, as longstanding readers of this blog know, homeopathy was “discovered” by a German physician named Samuel Hahnemann in the late 18th century, just 227 years ago, making me wonder if it could even qualify as “traditional” European medicine. Naturopathy, too, is European in origin, which alone doesn’t disqualify it as “traditional.” What does disqualify it as “traditional” is that it is an even more recent development than homeopathy, having arisen from the “natural cure” movement in Europe in the 1800s, and only having been named “naturopathy” in 1895, with the first school of naturopathy founded in 1901. As for osteopathy, it was invented in the US in 1874, founded by Andrew Taylor Still, a 19th-century American physician, Civil War surgeon, and Kansas state and territorial legislator.
As for norethindrone, this is an example that I had never seen advocates point to as validation of traditional medicine before, which is why I did a bit of Googling and PubMed searching. It turns out that this is just another example of the WHO citing a natural product that came to be used as a medicine as if that in itself validated “traditional medicine.” Norethindrone was indeed derived from two species of inedible wile Mexican yams, cabeza de negro (Dioscorea mexicana) and barbasco (D. composita), in the local language, and the use of the drug as birth control came about through nothing more remarkable (or “traditional”) than the application of pharmacognosy, the branch of pharmacology devoted to finding compounds in natural products that have medicinal properties.
The promotion continued:
WHO’s #TraditionalMedicine programme aims to build solid evidence base for policies and standards on traditional medicine practices and products, helping countries integrate it into their health systems & regulate its quality.
Here's how: https://t.co/Mld5hoIabv pic.twitter.com/wzaAXVNAHG
— World Health Organization (WHO) (@WHO) August 14, 2023
This represents yet another example of putting the cart before the horse. Note how the WHO doesn’t say that it will study traditional medicine to see if it works or to identify specific practices that might work. Rather, it assumes that traditional medicine works and promises to work to “build solid evidence base for policies and standards on traditional medicine practices and products, helping countries integrate it into their health systems & regulate its quality.”
Only after days of promotion, which began a week before the summit, does the WHO get around to caveats:
Listen to Dr Susan Wieland explain how to safely use #TraditionalMedicine.
Find out more in our FAQs:📌 https://t.co/BcDKRntETt
🔉 @lswieland pic.twitter.com/IccWYU792J
— World Health Organization (WHO) (@WHO) August 15, 2023
Out of curiosity, I looked at the FAQ referred to above and found more of the WHO putting the cart before the horse, for example:
A remarkable and rapid modernization of the ways traditional medicine is being studied can help realize the potential and promise of traditional medicine and traditional knowledge, for health and well-being. Taking clues from traditional uses, new clinically effective drugs can be identified through research methods such as ethnopharmacology and reverse pharmacology.
The application of new technologies in health and medicine can open new frontiers of knowledge on traditional medicine. Artificial intelligence (AI) has emerged as a game-changer, revolutionizing the study and practice of traditional healing systems. AI’s advanced algorithms and machine learning capabilities can allow researchers to explore extensive traditional medical knowledge, map evidence and identify once elusive trends.
AI…it had to be…AI. Seriously, just as adding AI to homeopathy doesn’t make it any less pseudoscientific (or ridiculous), adding AI to all of traditional medicine will not make it scientific. Notice the assumption, though. The WHO assumes that there is enormous promise in studying traditional medicine and combining it with science-based medicine. This proposition, as lawyers would say, assumes facts not in evidence, and even the WHO’s caveats about traditional medicine are the weakest of weak sauce. For example, I laughed out loud when the WHO asserted, seemingly reasonably, that natural products and traditional medicine practices “must pass the same level of scrutiny as modern pharmaceuticals.” Notice what precedes this statement:
#TraditionalMedicine practices and products must pass the same level of scrutiny as modern pharmaceuticals. 
Over 120 Member States have passed laws or regulations for herbal medicines, enhancing their safety and efficacy.https://t.co/nlTGS1BRYB pic.twitter.com/yIE6ofjah8
— World Health Organization (WHO) (@WHO) August 16, 2023
That’s right. Only right after pure boosterism for traditional medicine by an athlete does the WHO suggest that traditional medicine should be subject to the same scientific standards as science-based medicine. Here’s the problem. It already has been, for the most part, and been found wanting. I like to quote Dara ÓBriain on herbal medicine:
I’m sorry, ‘herbal medicine’, “Oh, herbal medicine’s been around for thousands of years!” Indeed it has, and then we tested it all, and the stuff that worked became ‘medicine’. And the rest of it is just a nice bowl of soup and some potpourri, so knock yourselves out.
Indeed. Or, to paraphrase an adage, “alternative medicine” that has been show through science to be effective and safe ceases to be “alternative” and becomes just “medicine.” In fairness, we “haven’t tested it all,” but it’s pretty unlikely that there’s much left untested that’s hiding any sort of amazing new treatments and cures, no matter how much the WHO wants this to be so or how much into a pretzel it has to twist the definition of “traditional medicine” in order to include other popular forms of alternative medicine, like naturopathy, homeopathy, and osteopathy. Moreover, any sort of medicine that includes homeopathy as potentially “science-based” is anything but science-based. I strongly suspect that the WHO included these European and American forms of quackery as being somehow “traditional medicine” in order to get European and American advocates of quackery on board with its summit.
A one-sided summit
But what about the summit itself? Let’s just say that my perusal and sampling of some of the talks showed it to be entirely consistent with the WHO’s promotion of traditional medicine and gauzy propaganda videos like this one:
Again, this video is propaganda, not reality.
It is not entirely unreasonable to hold a conference to examine traditional medicine practices and treatments in light of modern science. However, that is not what this is about. As Edzard Ernst was quoted saying in an article published in Nature, although he included his full statement (of which the Nature article only used a brief excerpt) on his blog:
The WHO has a long history of uncritically promoting alternative therapies. The Indian government has recently advocated irresponsibly dangerous nonsense, such as the use of homeopathy for the prevention and treatment of covid infections. The two together make an ominous initiative when it comes to alternative medicine.
Of course, there is nothing wrong in hosting a constructive dialogue about this subject. What seem ill-conceived, however, is the fact that the conference exclusively includes speakers who are staunch proponents of alternative medicine, a subject that, after all, remains highly controversial. Progress is not created by voicing one-sided, biased opinions. I fear that this meeting will result in the often before voiced platitudes and wishful thinking which no true scientist is then able to take seriously.
The Nature article only quoted the last sentence of this statement. Ernst, however, has nailed exactly the problem with this summit and pretty much every other conference held on “integrative medicine”: They are organized by believers in alternative medicine and include as speakers only believers, suffused with this “cart before the horse” mentality, for example, inadvertently described in the Nature article:
Lisa Susan Wieland, director of Cochrane Complementary Medicine at the University of Maryland School of Medicine in Baltimore, and an external adviser for the summit, says that the participants will discuss ways to gather evidence for traditional healing systems.
Wieland says that the quality and quantity of research for traditional medicine needs to improve before conclusive statements can be made about its safety and efficacy. “A lot has changed over the past 15 years,” she says. “Where there was previously insufficient good-quality research to determine what does and doesn’t work, we are now seeing more and better research on some traditional medicine.”
Cochrane Complementary Medicine? I hadn’t known before reading this that there was a Cochrane Complementary Medicine or that it was coordinated at the University of Maryland, an institution that has featured in this blog many times before since very early on for the uncritical promotion of “integrating” quackery into science-based medicine by its Integrative Medicine program. After having read this, I looked at the list of members of the advisory group for this summit. Every single member, from co-chairs Lisa Susan Weiland and Bhushan Patwardhan on down, holds a high-ranking faculty, advocacy, or government position in “integrative” or “traditional” medicine. You might think that if the WHO were truly interested in a science-based approach to studying “traditional medicine” it might include at least a token skeptic or two on the advisory group and among the speakers. It didn’t. That’s because the conclusions of the summit were foreordained before it was even convened.
Again, this is not about using science to study what does and doesn’t work in “traditional medicine.” It is about assuming that traditional medicine is of immense value on par with that of science-based medicine and then looking for evidence to support that conclusion. Again, it’s about putting the cart before the horse, as evidenced by just the titles and descriptions of some of the sessions, such as “Planetary health and well-being for all: Learning from Indigenous knowledges, and traditional, complementary and integrative health systems,” “Integrative health for well—being at all ages,” and “Contribution of Traditional Medicine to Healthcare Systems.” (Also, note the titles of many WHO press releases on the conference, like WHO summit aims to unlock power of traditional medicine through science. and describing its Global Centre for Traditional Medicine as Catalysing ancient wisdom and modern science for the health of people and the planet.) Again, note the assumptions, namely that traditional medicine is an important part of “planetary health and well-being for all” that we have to learn from and that traditional medicine has already made great contributions to the healthcare systems of many nations, if only we would just recognize that. (And, make no mistake, the purpose of this summit is to promote that idea as justification not just for the $250 million spent to found the WHO’s Global Centre for Traditional Medicine but for the continued promotion of traditional medicine and its “integration”—whether science-based practitioners like it or want it or not—into the global health system through the ICD-11 and other means.
Unsurprisingly, the day after the summit ended, the WHO released a statement, Global partners commit to advance evidence-based traditional, complementary and integrative medicine, which concluded pretty much what I expected the WHO to conclude from the day that I learned about this summit, which now seems to have renamed “traditional medicine” to “traditional, complementary, and integrative medicine” (TCIM):
Preliminary findings from the WHO Global Survey on Traditional Medicine 2023 shared at the Summit indicate that around 100 countries have TCIM related national policies and strategies. In many WHO Member States, TCIM treatments are part of the essential medicine lists, essential health service packages, and are covered by national health insurance schemes. A large majority of people seek traditional, complementary and integrative medicine interventions for treatment, prevention and management of noncommunicable diseases, palliative care and rehabilitation.
Dr Bruce Aylward, Assistant Director-General, Universal Health Coverage, Life Course at WHO, highlighted the need for a “stronger evidence base—a WHO priority—to enable countries to develop appropriate regulations and policies around traditional, complementary, and integrative medicine.”
Tell me something I didn’t know 15 years ago, when this blog was founded. Advocates of unproven and disproven treatments always claim that there is “insufficient evidence” and that we need a “stronger evidence base.” The latter statement is one of those vague generalities that is always true to one degree or another. It is something that believers in prescientific and pseudoscientific traditional medicine approachers were saying 20 years ago, when I first became heavily involved in assessing “nontraditional” health claims, that they continue to argue now, and that they will likely continue to argue long after I have shuffled off this mortal coil.
It is thus utterly unsurprising that the WHO press release after the conference concludes:
In closing the Summit, Dr Hans Kluge, WHO Regional Director for Europe said, “Together, we have gently shaken up the status quo that has, for far too long, separated different approaches to medicine and health. By taking aim at silos, we are saying we will collaborate all the more to find optimal ways to bring traditional, complementary and integrative medicine well under the umbrella of primary health care and universal health coverage.” He further added, “We have reiterated how crucial it is to get better evidence on the effectiveness, safety and quality of traditional and complementary medicine. That means innovative methodologies for assessing and evaluating outcomes.”
Dr Shyama Kuruvilla, Senior Strategic Adviser and lead for the WHO Traditional Medicine Global Centre, who also led the organization of the Summit said, “We learnt much about the existing policies, tools and practices. But it is clear we have a long journey ahead in using science to further understand, develop and deliver the full potential of TCIM approaches to improve people’s health and well-being in harmony with the planet that sustains us.”
The Summit’s summary document included conclusions and commitments from participants on wide-ranging issues, from global policy, leadership, innovation, health workforce, data, evidence, monitoring, regulation, legal frameworks and protecting biodiversity and sustainable development.
The WHO and other believers in traditional medicine are now touting the Gujarat Declaration as the key to “integrating” traditional medicine into medicine going forward. Of note, the Gujarat Declaration was referenced in the opening remarks by WHO Director-General Tedros Adhanom Ghebreyesus:
This summit is an important opportunity to advance the understanding and use of traditional medicine.
The Gujarat Declaration—the main outcome of this Global Summit—if effectively implemented, will enhance the appropriate integration of traditional medicine into national health systems.
Let me leave you with three specific requests.
First, we urge all countries to commit to examining how best to integrate traditional and complementary medicine into their national health systems.
Second, I urge you all to identify specific, evidence-based and actionable recommendations that can inform the next WHO traditional medicine global strategy.
Third, I urge you to use this meeting as the starting point for a global movement to unlock the power of traditional medicine through science and innovation.
Once again, my thanks to India for its hospitality and leadership in this area.
Again, note how the purpose of this summit was not to determine whether traditional medicine has value and to demonstrate what does and doesn’t work. Rather, the purpose was to promote the “integration” of “traditional medicine” into science-based medicine and to go looking for evidence to justify that predetermined goal.
Let me suggest something that might seem a bit heretical to the WHO as it exists now. The $250 million spent on its quackery-promoting Traditional Medicine Global Centre would be much better spent on—dare I say?—more traditional public health programs in areas of the world that desperately need them, programs promoting clean water, vaccination, and better nutrition, as well as providing “on-the-ground” public health workers and medical personnel to bring science-based medicine, rather than unproven woo, to the people who need it most. The WHO has done such good work in these areas over the decades, and it saddens me to see it slipping more and more into a mindset that promotes medicine based on mysticism and prescientific religious belief systems, such as TCM and Ayurveda, as somehow being co-equal with science-based medicine to the point that they should be “integrated” with it. That is not promoting “planetary health and well-being for all.” Rather, it is shunting precious global health care funds away from science-based public health and medicine to promote unproven and disproven “ancient knowledge.”

Science Moms Fight Fears with Facts
A new documentary takes a novel approach. It features scientist moms who are just like other moms except that they understand the science. They set the record straight about GMOs, vaccines, and other subjects of interest to parents. They provide the facts to counteract unreasonable fears.
Harriet Hall on November 28, 2017
Link

At the recent conference of the Committee for Skeptical Inquiry (CSICON) in Las Vegas, on October 28, 2017, I had the great privilege and pleasure of being in the audience for the American premiere of a new documentary, Science Moms, as well as for the following live panel discussion by the women featured in the movie. In the documentary, a group of scientists and science communicators who are also moms address misperceptions created by misinformation in the media about GMOs, vaccines, and other issues important to parents. They point out that “moms whose opinions are formed by fear and hype are so loud. But they’re the only people talking about it, the only resource people have.” With this documentary, people now have another resource based on science, a resource that is easily digestible and compelling.
The film starts with a beautiful sunrise and a Gwyneth Paltrow quote: “The sun is the sun – how can it be bad for you? I don’t think anything that’s natural can be bad for you.” The Moms answer:
“Wow! I could make a list for her.”
“The sun causes cancer.”
“Nature will kill you, really quickly.”
“Sometimes I think she’s trolling us.”

Next, the Science Moms are introduced and talk about how they got interested in science. They are:
    • Anastasia Bodnar, PhD, Plant Geneticist
    • Alison Bernstein, PhD, Neuroscientist
    • Layla Katiraee, PhD, Molecular Geneticist
    • Jenny Splitter, Science Communicator and Storyteller
    • Kavin Senapathy, Science Communicator
These women shatter the stereotypes of scientists as commonly portrayed in the media. They are normal, friendly, personable, attractive, well-groomed, non-geeky, everyday people, just like other working moms except that their jobs happen to involve science. Moms viewing the film ought to be able to relate to them and listen to what they have to say just as they would listen to their friends.
Two of Science Moms were fans of Buffy the Vampire Slayer and were appalled to learn that the actress who played Buffy, Sarah Michelle Gellar, was speaking out against GMOs. They joined a group of 15 women scientists, bloggers, and educators to send a letter to Paltrow, Gellar, and other celebrities asking them not to co-opt motherhood and wield their fame to oppose beneficial technologies, but to use their influence responsibly and ensure that their advocacy is supported by facts, not fear.
The letter caught the attention of Natalie Newell, the host of “The Science Enthusiast” podcast. She contacted one of the letter’s authors. One thing led to another, and the result was this documentary.
The Moms acknowledge that being a parent is scary. Parents desperately want to protect their children from any possible harm, and often they aren’t sure how to best do that. Even they admit to having acted irrationally based on unrealistic fears for the welfare of their children. It’s a great marketing technique: “If you can scare a parent, of course they’re going to shell out for the alternative.”
GMOs
People who don’t know anything about GMOS may choose organic because they vaguely remember hearing that it was better for their kids. GMOs are presented in the media as inserting genes of one species into another species. But that’s only one meaning. Genetic modification also means selective breeding, cross breeding, mutagenesis, genome editing, and other techniques.
When plants are cross-pollinated, a gene for disease resistance can spread to another species, but that’s random. Why not use technology to put the desired gene into the plant? In reality, almost everything we eat was genetically engineered centuries ago by our ancestors’ selective farming and breeding practices.
The Moms point out these benefits of genetic modification:
    • Drought resistance
    • Pest resistance
    • Disease resistance
    • Increased crop yield
    • Increased nutritional content
    • Economic benefits
    • Reduced pesticide usage
    • Reduced greenhouse gas emissions
Vitamin A deficiency causes untold cases of blindness and death in developing countries. Golden rice was genetically modified to supply vitamin A, but thanks to anti-GMO ideology it hasn’t reached those who need it most.
Some people fear that eating something genetically engineered will genetically engineer THEM. Not hardly! Despite widespread fears, GMOs have never harmed a single person’s health in any way.
Fear of chemicals
The idea that “There is no safe amount of chemicals” is false. Everything is made of chemicals. They show a long list of all the scary-sounding chemicals in an all-natural blueberry. Pears naturally make formaldehyde.
The “most brilliant marketing move of the last ten years” was to convince everyone that organic is pesticide free. Copper sulfate is really bad for the environment, and it’s allowed in organic farming.
Data doesn’t support claims that organic is pesticide free, better for environment, or healthier.
There are no health benefits to be gained from organic. It’s just more expensive.
Vaccines
We hear:
    • Too many too soon
    • Dangerous chemicals in vaccines
    • I prefer to fight off disease naturally
    • It’s a Big Pharma conspiracy
    • “These diseases aren’t really that dangerous”
None of these are based on evidence or science. Unrealistic fears of vaccines have led to decreased herd immunity and disease outbreaks. Our grandparents aren’t likely to fear vaccines, because they knew people who died of polio and other preventable diseases. It’s ironic that people are afraid of harmless GMOs but don’t fear the very real risks of vaccine preventable diseases.
Homeopathy
One Science Mom says, “I’m embarrassed to say I tried it. When I found out what it was, I thought ‘Oh, that’s why it didn’t work.’ I could have given the kids sugar water I made at home and saved a few bucks.”
I can’t imagine parents reaching for something that is untested, unregulated, and has no active ingredients in it. It baffles me.
Perhaps it’s because people want to do things on their own – homeopathy, homemade baby formula, anything that gives them the illusion of being in control.
Who’s paying you??!!
The answer to this oft-repeated question is an emphatic “Nobody!” Kavin Senapathy says she has been called a fake mom, has gotten death threats, and has been told her name is made up (as if Monsanto would invent a name like Kavin Senapathy!) She doesn’t understand where the shill accusation comes from. The assumption seems to be that anyone who doesn’t have the same world view as you, must be paid to have that view. It’s hard to have your world view challenged, so it’s easier to think they must be paid to disagree with you than to think your world view might be incorrect.
More
They explain that scientific consensus is not like a vote, it’s the confluence of all the evidence coming together around a hypothesis.
When people ask if something is safe for their child, the best advice is to go to a real doctor (not a naturopath); and to buy real medicine (homeopathy is not real medicine).
Healthy diet? Eat lots of fruits and vegetables, buy whatever’s cheaper, wash produce.
Some organizations are trying to scare people away from buying certain fruits and vegetables. That’s CRAZY!
You might as well enjoy being a parent. “Basic safety stuff fits on half a page.” Don’t worry about minor details with no solid evidence, like when to introduce solid foods.
“When kids are 10-12, no one’s talking about whether they were breast fed.” The effects of stress on us and our kids are way worse than anything we’re worrying about.
What’s the real issue? If it’s corporate control of our political system, that’s a valid concern that many of us share. But GMOs aren’t the cause of that. Focus on the real source of the anger rather than blaming a proxy.
Fear-based communities bring people together. The Science Moms are trying to create a new community based on science and reason; based on facts, not fear.
Conclusion: A lot of people really need to watch this documentary
Science Moms is short and to the point. The 30-minute film is scientifically accurate, persuasive, and well-designed, with good production values. It’s available online for purchase at $4.99. I hope it will be more widely disseminated, because it offers important information that the general public needs to hear. People who have been exposed to anti-GMO or anti-vaccine propaganda are not likely to seek out, read, and understand the scientific evidence. But perhaps they will be willing to listen to moms who are just like them but who have the advantage of understanding the science.

A Harris Poll on “Alternative Medicine”
Steven Novella on May 11, 2016
Link


Mark Twain popularized the phrase, “There are three kinds of lies: lies, damned lies, and polls and surveys.” (He may have said “statistics” at the end, but I think this version works as well.)
A new Harris Poll on “alternative medicine” nicely demonstrates some of the problems with polls. The biggest problem is how you frame the questions. You can dramatically affect the results of the poll by exactly how a question is phrased, which other questions come before or after the question, and the overall context of the poll.
In essence a poll is a two-way form of communication. While it is meant to derive information from the subjects of the poll, those subjects are also receiving information from the poll itself. That is the very reason for “push polling” – which is the practice of disguising campaigning messages as a poll. (“On a scale from 1-5, how much are you bothered by the fact that candidate X is a misogynist?”). Yes, Prime Minister has an amusing example of this practice in action.
Interpreting the results of a poll is also not straightforward. You have to know how the subjects of the poll interpreted the questions, and what factors might affect their response. 
The Harris Poll
Polls looking into attitudes and use of so-called “alternative medicine” are automatically tricky because the definition of “alternative medicine” is deliberately vague. What counts as alternative? What is the operational definition?
I have argued that it is a false category, designed for marketing and propaganda purposes. That is why proponents have been shifting the phrase over the years, from alternative to complementary, and now to integrative medicine. In practice it seems to include anything that is not adequately based on science and evidence, plus anything else that proponents want to include to make the category seem more popular and legitimate.
The biggest weakness of the poll is that it makes no attempt to define “alternative.” Does this include exercise, taking vitamins, prayer, getting a massage, or relaxation?
Further, it contrasts “alternative medicine” with “conventional Western medicine.” This is buying into the false dichotomy even further. Calling science-based medicine “Western” is an insult to our colleagues in the East who are just as capable and dedicated to science as their Western counterparts.
While the poll does not define the key concept of the entire poll, it does suggest to those taking the poll what might be included in the definition by asking them about specific alleged “alternative” interventions.
Here is a quick summary of the key results from their press release:
Overall, about seven in ten Americans (71%) have used some kind of alternative therapy before. Over one third state they have used herbal medicines (37%) or chiropractic care (34%). For Matures (those aged 70+), chiropractic care is the predominant type of alternative therapy used, with nearly half having ever used it (45%). Herbal medicines appeal much more to a younger audience, with 40% of those under 50 having used it.
Usage of alternative therapies is higher among those who don’t have insurance – nearly half of uninsured Americans (48%) use alternative therapies as often as or more than conventional treatments.
When it comes to how likely Americans are to use alternative options for various conditions, a majority of all adults (64%) are likely to use these options to treat physical pain, and Millennials and Gen Xers are particularly likely to consider such alternatives for things like addiction (60% and 57%, respectively) and mental health (57% each) as well.
The 71% is a typical inflated statistic of alternative medicine use when using the broadest definition. If you dig a little deeper into the poll you see that this is mainly comprised of chiropractic, supplements/vitamins, and massage. Aromatherapy and meditation also are over 20%.
What does it mean?
Chiropractic is always near the top because it has been institutionalized in the US for a century now. Some patients use chiropractors as their physical therapists. Chiropractors also practice a lot of blatant pseudoscience, making this also a complex category.
Generations of aggressive marketing and the deregulation of supplements in 1994 has also exploded the supplement market, so again, no surprise there. The popularity of aromatherapy and essential oils is a bit of a recent phenomenon, and to me is mostly an extension of the supplement phenomenon.
Massage has always been popular, and is a perfectly reasonable intervention for muscle tension. It is a great example of an intervention that is not really alternative, unless you make pseudoscientific claims for its effects (such as squeezing out toxins from the tissues).
Meditation, in my opinion, is relaxation. Studies looking into its benefits simply show the benefits of relaxation, and often make no attempt to compare meditation to other forms of relaxation, and so do not establish that there is any specific benefit to meditation itself.
As you can see, the category is full of sloppy thinking, loose definitions, and dubious science. But, as you can also see, the vast majority of what people are using under the label of “alternative medicine” comes down to essentially some kind of supplements, physical therapy, or relaxation.
When we get to what I consider the hard core “alternative” treatments, ones that are not just a rebranding of lifestyle factors, diet, and physical therapy, the numbers drop to very low levels. The most popular such modality in this poll barely cracks double digits – acupuncture at 11%. The rest are all single digits.
This is also not surprising given the degree to which the scientific research on acupuncture has been systematically misreported to the public. What the research actually show, quite clearly, and consistently after thousands of clinical trials, is that acupuncture simply does not work, for anything. If you define acupuncture as sticking thin needles into acupuncture points to address specific symptoms or health issues, then acupuncture does not work. The evidence clearly shows it does not matter where you stick the needles, or even if you stick the needles. Efficacy depends only on the ritual surrounding acupuncture, and having a positive interaction with the acupuncturists. In other words, acupuncture is an elaborate placebo, which is a scientific way of saying it does not work.
Meanwhile, the same research which shows it does not work is systematically reported to the public (and presented to academics and professionals who apparently fail to do their due diligence), as if it does work. They essentially say that placebo acupuncture works also. This is identical to a pharmaceutical company arguing that the placebo of their drug also works to relieve subjective symptoms, and therefore their drug works (because there was no difference between the two).
All of this deception has pushed acupuncture to 11% (at least in this poll) in terms of people saying they have ever used it. From there it goes: electrotherapy (also vague) – 9%, reflexology – 5%, hypnotherapy – 4%, Reiki – 3%, cupping – 3%, other – 3%. Homeopathy did not even make the cut, apparently.
It is also important to note what the poll does not ask. For example, a previous survey found that the phrase, “what is proven to work best” was the highest ranked, at 79%, as a description of what patients want in their medical treatments.
The public does not appear to be turning away from science, or from science-based standards of care. They want their treatments to be safe and effective. They are simply being misinformed about what this means. They are often being lied to and directly deceived, often with an immediate profit motive on the part of those selling snake oil.
Conclusion: PR, not polling
This Harris poll on “alternative medicine” is so terrible it is practically a pro-alternative medicine push poll. They do not define their key terms, use loaded language and false dichotomies, and use vague categories.
I should also point out that this was an online poll of those who agreed to take the poll. The results can therefore not be generalized to the population as a whole. It is quite possible that people who are frequently online are more exposed to pro-alternative medicine marketing. Those willing to take the poll may also be more pro-alternative medicine.
Even without these weaknesses, the design of the poll itself was very poor, in my opinion. The reporting of the results was also highly biased, reflecting the bias in constructing the poll itself.
This poll represents one piece of the big picture – that of a self-fulfilling narrative. Those with an ideological agenda, or who make money by selling dubious treatments to the public, have been engaged in a propaganda campaign to rebrand what was previously known as health fraud as if they were a legitimate alternative. This campaign, unfortunately, has been very successful. They have controlled the narrative to the point that it is not even questioned, except by those who are dedicated to high standards of science within health care.

On TikTok, Everything Is a Poison or Superfood
Steven Novella on March 19, 2025
Link
We have to face the fact that social media is now the primary way that many people get their information. More than half of people in the US get their “news” from social media at least some of the time. But I don’t think asking people where they get “news” captures the full phenomenon. People who watch TikTok videos might not consider that consuming news.
By analogy, people probably don’t consider watching sitcoms or soap operas consuming news, but evidence consistently shows that such exposure to pop culture influences our beliefs and attitudes. So, like it or not, we have to consider the possibility that our culture, to a significant extent, belongs to social media now. This has far reaching implications for our society, but I want to consider just one – beliefs and attitudes toward medicine and health.
I have been promoting SBM on YouTube, Facebook, and more recently TikTok, and the environment is grim. There are pockets of sanity, and good science communicators doing their best, but I have noticed some disturbingly common beliefs. The level of cynicism and conspiracy-mongering is profound. Anti-intellectualism, anti-institutionalism, and anti-expertiseism (I think I just made that word up) is rampant. I try to whack several moles a week on TikTok, but I am just scratching the surface.
Focusing a bit further, one specific attitude I find extremely common is notion that everything is either a horrible toxin or a life-saver. While this attitude is part of the online zeitgeist, it is also deliberately cultivated by self-proclaimed health gurus and influencers. This predates social media, but is now far more prevalent.
Everyone with a cell phone, regardless of their background, education, and experience, can post on TikTok that some everyday food is a horrible toxin and you should just throw it away. Don’t eat peanut butter, or spinach, and don’t use tea bags or toilet paper. The list is endless. Often they are just promoting their own health-guru brand, so they have to have something shocking and scary to tell you. They may also be promoting their own alternative products. But often they are just falling victim to chemophobia – paying forward the same nonsense they are consuming.
It is so rampant that you will see in the comments users noticing how out-of-control it is. “So we can’t eat anything now” is a common observation. It would be interesting for someone with far more time than I currently have to systematically go through a list of common food items and ingredients and see how many of them are targeted by some TikToker who says it will kill you.
The underlying assumptions here, which can either be implicit or explicit, include the notion that the regulatory agencies are failing you. The FDA, USDA, and EPA, apparently, are unaware of all these stunning facts or are in the pocket of industry or simply don’t care. Doctors who are not themselves online gurus apparently also don’t know health science or are part of some sinister conspiracy. This is just taken as a given.
Since I can’t (in one lifetime) take on all this nonsense individual, I do try to spread some healthy attitudes to protect against TikTok health doomscrolling.
First, stop with the black-and-white false dichotomy thinking. Most foods are neither horrible toxins or superfoods – they are just food. Eat a varied diet with plenty of plant matter, and everything in moderation, and chances are you’ll be fine. Unless you have a specific medical condition that requires special attention, for most people that is a good first approximation of a healthful diet. Any more detail is unlikely to provide a statistically significant benefit, or be worth the effort or expense. In fact, avoid lots of different foods, going on special diets, is far more likely to be harmful than helpful.
Related to this is the foundational concept of toxicology – the dose makes the poison. Everything is a poison in a high enough dose, and most things are safe in a low enough dose. You can be sure that anything that is not safe in any dose is already banned or there are efforts to minimize human exposure.
Everything is a chemical. There are chemicals in every single thing you have ever consumed. Water is a chemical. Bananas are loaded with chemicals. The fact that something contains chemicals, in and of itself, is meaningless. What matters is – which chemicals and at what dose.
“Natural” is meaningless. That something occurs in nature does not mean it is harmless or helpful. Nature does not care about you. In fact, at a first approximation, nature is actively trying to kill you. Plants make toxins to keep you from eating them. Animals make toxins to kill you. That is why almost every plant you eat has been cultivated beyond recognition to be human-friendly. There is also nothing inherently unhealthy or risky about a chemical that is manufactured or synthetic. All that matters is the science and the evidence for safety at various doses.
Don’t just assume there is a ridiculous conspiracy. This is a big topic, but just remember not to be gullible when someone claims there is a giant conspiracy. Yes, there are some psychopaths out there, and corporations will look out for their own interests, but there are not entire institutions full of supervillains. No one is hiding the cure for cancer. There isn’t some black ops organization that will swoop in and kill someone who cures a disease or invents something truly useful.
You are not as smart as you think you are. This is true of everyone – of me, of experts, of professionals, and the public at large. The world is a big and extremely complicated place, and science is hard. So show a little humility. The chance that you have pierced the veil, that you have seen with clarity what literally billions of people, and millions of experts, have missed is vanishingly small. If something does not make sense to you, don’t assume the world it broken or evil or conspiring against you. You probably just made a mistake, or don’t know something.
This is why we need to leverage institutions, processes, and crowdsourcing. We need to do careful study and observation, and have experienced people poor over and debate what those studies find, and then put the results through a tested process of regulation that accounts for all possible risks and benefit. It’s time consuming and complex, but that’s the world.
If you think you are so smart that you can give better health advice that millions of experts pooling their expertise, the chancing are overwhelming that you are just guilty of hubris and are spouting nonsense.
This doesn’t mean the experts always get it right. Of course they don’t – as I said, this stuff is complicated, and we don’t know everything. That is also an unreasonable standard, and not a reason to listen to self-proclaimed gurus shooting from the hip. Experts and institutions make mistakes, but there is a process to correct those mistakes, to do things a little better going forward.
So go watch your cat videos – but I would not recommend getting your health advice from randos on social media.

